Patent Application
20210014513
The present invention relates to the field of encoding/decoding pictures of a video. Embodiments concern an improvement in encoding/decoding pictures of a video to accelerate encoding/decoding of pictures, such as by performing specific processes in encoding/decoding pictures of a video in parallel.
Video compression algorithms used in encoding pictures of a video become more and more sophisticated and new features gaining higher compression ratios are often directly coupled to additional computational complexity. To compensate this increase in complexity and make applications using advanced video compression tools faster and usable, efforts have to be taken.
Two different approaches are at hand to accelerate a video encoder. One deals with algorithmic optimization, including the pruning of encoding search space. The other utilizes different techniques of programming and execution without altering the algorithm as such.
Accordingly, there is a need for approaches that allow for reduction of the time necessary for encoding and/or decoding pictures of a video.
Embodiments of the present invention provide for such approaches.
An embodiment may have a video encoder for encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, wherein the video encoder is configured to: by a first thread, process coding blocks in a first line of the array of coding blocks; by a second thread, process coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; upon a specific event, interrupt processing coding blocks in the second line of the array by the second thread and process coding blocks of a third line of the array by the second thread, wherein a) the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in the second line, or b) the specific event is that processing the second line reaches a specific coding block in the second line, the specific coding block representing a line switch barrier, and that the second line is the last line currently processed by one of the threads of the video encoder, wherein upon reaching the line switch barrier, the second thread jumps to the first coding block of the third line, which is the next line following the second line, to process the first coding block of the third line.
Another embodiment may have a video decoder for decoding a data stream into pictures of a video, the picture being partitioned into an array of coding blocks, wherein the video decoder is configured to: by a first thread, process coding blocks in a first line of the array of coding blocks; by a second thread, process coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; upon a specific event, interrupt processing of coding blocks in the second line of the array by the second thread and process coding blocks of a third line of the array by the second thread, wherein a) the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in the second line, or b) the specific event is that processing the second line reaches a specific coding block in the second line, the specific coding block representing a line switch barrier, and that the second line is the last line currently processed by one of the threads of the video encoder, wherein upon reaching the line switch barrier, the second thread jumps to the first coding block of the third line, which is the next line following the second line, to process the first coding block of the third line.
Another embodiment may have a method of encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, the method having the steps of: by a first thread, processing coding blocks in a first line of the array of coding blocks; by a second thread, processing coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; upon a specific event, interrupting processing coding blocks in the second line of the array by the second thread and processing coding blocks of a third line of the array by the second thread, wherein a) the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in the second line, or b) the specific event is that processing the second line reaches a specific coding block in the second line, the specific coding block representing a line switch barrier, and that the second line is the last line currently processed by one of the threads of the video encoder, wherein upon reaching the line switch barrier, the second thread jumps to the first coding block of the third line, which is the next line following the second line, to process the first coding block of the third line.
Another embodiment may have a method of decoding a data stream into pictures of a video, the picture being partitioned into an array of coding blocks, the method having the steps of: by a first thread, processing coding blocks in a first line of the array of coding blocks; by a second thread, processing coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; upon a specific event, interrupting processing of coding blocks in the second line of the array by the second thread and processing coding blocks of a third line of the array by the second thread, wherein a) the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in the second line, or b) the specific event is that processing the second line reaches a specific coding block in the second line, the specific coding block representing a line switch barrier, and that the second line is the last line currently processed by one of the threads of the video encoder, wherein upon reaching the line switch barrier, the second thread jumps to the first coding block of the third line, which is the next line following the second line, to process the first coding block of the third line.
Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method of encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, the method having the steps of: by a first thread, processing coding blocks in a first line of the array of coding blocks; by a second thread, processing coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; upon a specific event, interrupting processing coding blocks in the second line of the array by the second thread and processing coding blocks of a third line of the array by the second thread, wherein a) the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in the second line, or b) the specific event is that processing the second line reaches a specific coding block in the second line, the specific coding block representing a line switch barrier, and that the second line is the last line currently processed by one of the threads of the video encoder, wherein upon reaching the line switch barrier, the second thread jumps to the first coding block of the third line, which is the next line following the second line, to process the first coding block of the third line, when said computer program is run by a computer.
Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method of decoding a data stream into pictures of a video, the picture being partitioned into an array of coding blocks, the method having the steps of: by a first thread, processing coding blocks in a first line of the array of coding blocks; by a second thread, processing coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; upon a specific event, interrupting processing of coding blocks in the second line of the array by the second thread and processing coding blocks of a third line of the array by the second thread, wherein a) the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in the second line, or b) the specific event is that processing the second line reaches a specific coding block in the second line, the specific coding block representing a line switch barrier, and that the second line is the last line currently processed by one of the threads of the video encoder, wherein upon reaching the line switch barrier, the second thread jumps to the first coding block of the third line, which is the next line following the second line, to process the first coding block of the third line, when said computer program is run by a computer.
According to an embodiment of the invention, a video encoder for encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, is configured to, by a first thread, process coding blocks in a first line of the array of coding blocks. The video encoder is further configured to, by a second thread, process coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed. The video encoder is configured to, upon a specific event, interrupt processing coding blocks in the second line of the array by the second thread and process coding blocks of a third line of the array by the second thread.
Thus, according to embodiments of the present invention, the second thread may switch to a different line of the array of coding blocks before processing of the line it is actually processing is finished, which allows a higher flexibility and a reduction of idle times.
In embodiments, the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in second line. Thus, idle times of the second thread may be reduced since the second thread does not have to wait until the first thread has finished processing of a specific block in the first line, but may switch to another line of the array of coding blocks to process coding blocks in this line, rather than being not in operation until the first thread has finished processing the useful coding blocks.
In embodiments, the specific event is that processing the second line reaches a specific coding block in the second line. Thus, embodiments permit coding the first line of a next line of coding blocks in advance. This may help preventing possible blocking of processing of the next and the next but one line if processing is to switch to the next and the next but one line in a reduced time frame. In embodiments, the coding blocks are grouped into tiles and wherein the specific coding block is not at a tile border. Thus, in embodiments, processing the first coding block in a subsequent line may be achieved in a flexible manner at an appropriate point in time.
Embodiments of the present invention provide for a video decoder for decoding a data stream into pictures of a video, the picture being partitioned into an array of coding blocks, wherein the video decoder is configured to: by a first thread, process coding blocks in a first line of the array of coding blocks; by a second thread, process coding blocks in a second line of the array while processing the coding blocks of the first line of the array is still in progress, wherein processing a coding block in the second line of the array involves that reference coding blocks in the first line have been processed; and upon a specific event, interrupt processing of coding blocks in the second line of the array by the second thread and process coding blocks of a third line of the array by the second thread.
In embodiments of the video decoder, the specific event is that the first thread did not yet finish processing the reference coding blocks in the first line that may be used for the second thread to process a next coding block in second line. In embodiments of the video decoder, the specific event is that processing the second line reaches a specific coding block in the second line. In embodiments of the video decoder, the coding blocks are grouped into tiles, wherein the specific coding block is not at a tile border.
Embodiments of the present invention provide for corresponding methods of encoding and decoding pictures of videos by processing lines of an array of coding blocks in parallel.
Embodiments of the present invention provide for a video encoder for encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, wherein the video encoder is configured to recursively partition coding blocks into smaller sub-blocks, wherein the video encoder is configured to: evaluate for each block or sub-block that may be further partitioned different split options, wherein, for a block or sub-block, which meets a specific criterion, at least two of the different split options are evaluated in parallel by separate threads.
Thus, embodiments of the present invention permit specific tasks to be performed in parallel. It was found that evaluating different split options in parallel is effective to reduce the time that may be used for encoding if a specific criterion is met. In embodiments, the specific criterion is that the block or sub-block is a sub-block and has a specific block size or less. In embodiments, the specific block size is less than the block size of the coding blocks, which may be CTUs (coding tree units). In embodiments, the specific block size is a size of 16 times 16 pixels. In embodiments, the specific criterion is that a quad split is not available for the block or sub-block.
In embodiments, the video encoder is to determine whether the block or sub-block meets the specific criterion, to evaluate the split options successively by one thread if the block or sub-block does not meet the specific criterion, and to evaluate the split options in parallel by the separate threads if the block or sub-block meets the specific criterion Thus, according to embodiments of the inventive approach, the additional resources that may be used for evaluating the different split options in parallel may be used in an effective manner.
In embodiments, the video encoder is configured to generate for each split option a separate thread. In embodiments, the video encoder is configured to group the different split options into sets and to generate for each set of split options a separate thread. Thus, embodiments of the invention permit grouping sets of split options in such a manner that the processing times of the evaluations performed in the separate threads may be balanced.
In embodiments, the different split options include at least two options of: a quad split partitioning the block or sub-block into four sub-blocks of the same size, a quad split partitioning the block or sub-block into four sub-blocks of different sizes, a horizontal bi-split partitioning the block into two sub-blocks of the same size, a vertical bi-split partitioning the block or sub-block into two sub-blocks of the same size, a horizontal bi-split partitioning the block or sub-block into two sub-blocks of different sizes, a horizontal bi-split partitioning the block or sub-block into two sub-blocks of different sizes, a vertical triple split partitioning the block or sub-block into three sub-blocks of the same size, a vertical triple split partitioning the block or sub-block into three sub-blocks of which at least two have different sizes, a horizontal triple split partitioning the block or sub-block into three sub-blocks of the same size, and a horizontal triple split partitioning the block or sub-block into three sub-blocks of which at least two have different sizes. In embodiments, the different split options include split options of the same type, wherein the sizes of the sub-blocks resulting from one of the split options of the same type are different from the sizes resulting from another one of the split options of the same type. Thus, embodiments permit for parallelizing the evaluation of a number of different split options for sub-blocks meeting a specific criterion.
Embodiments provide for corresponding methods of encoding pictures of a video into a data stream by evaluating split options in parallel.
Embodiments provide for a video encoder for encoding a series of pictures of a video into a data stream, wherein the series of pictures comprises pictures representing random access points, RAP-pictures, and non-random access point pictures, non-RAP-pictures. In order to encode non-RAP-pictures between two RAP-pictures, information concerning the two RAP-pictures is necessary. The video encoder is configured to provide data sub-streams corresponding to different sub-series of the series of pictures by different encoding units, each sub-series including at least one RAP-picture and the non-RAP-pictures between the one RAP-picture and the preceding RAP-picture. The encoding units are configured to encode the non-RAP-pictures of a sub-series of pictures using information concerning at least one of the two RAP-pictures adjoining the non-RAP-pictures, which is obtained by decoding an encoded version of this RAP-picture, which has been encoded by another encoding unit.
In embodiments of a video encoder as described in the preceding paragraph, the other encoding unit is a dedicated RAP-picture encoding unit configured to encode the RAP-pictures of the series of pictures, wherein the encoding units are configured to encode the non-RAP-pictures of a sub-series of pictures using the information concerning both RAP-pictures adjoining the non-RAP-pictures, which is obtained by decoding encoded versions of both RAP-pictures, which have been encoded by the dedicated RAP-picture encoding unit. In other embodiments of a video encoder as described in the preceding paragraph, each of the different encoding units is configured to encode the adjoining RAP-picture preceding the non-RAP-pictures of the sub-series of pictures and to obtain the encoded version of the adjoining RAP-picture following the non-RAP-pictures from the encoding unit providing a data sub-stream corresponding to an immediately following sub-series of pictures. In embodiments of the video encoder described in the preceding paragraph, the encoding units have an embedded decoder configured to decode the encoded versions of the RAP-pictures and to derive the information necessary to encode the non-RAP-pictures. In embodiments, the encoding units are configured to provide several data sub-streams in a timely overlapping manner.
Such embodiments of the present invention permit sub-series of pictures to be encoded in parallel by different encoding units, such as different threads, without having to encode both RAP-pictures or each series of pictures. Each RAP-picture has to be encoded only once, either by a dedicated RAP-picture encoding unit or by one of the encoding units. Thus, redundant encoding of RAP-pictures can be avoided. This is beneficial in view of the fact that encoding RAP-pictures is generally more time consuming than encoding non-RAP-pictures. In embodiments, the series of pictures may be a closed GOP (group of pictures) or an open GOP.
Embodiments provide for corresponding methods of encoding a series of pictures of a video, which comprises RAP-pictures and non-RAP-pictures into a data stream.
Embodiments provide a video encoder, which comprises an encoding unit configured to encode a series of pictures of a video to generate a data stream and a decoding unit configured to decode the generated data stream and to provide resume information concerning the decoded pictures. The video encoder is configured to, upon an interruption of an encoding process of the encoding unit at a specific picture of the series of pictures, provide the encoding unit with the resume information from the decoding unit to resume encoding directly at the specific picture without having to encode the pictures preceding the specific picture again. In embodiments of such a video encoder, the encoding unit is provided only with the resuming information concerning decoded pictures of the same intra-period, which are that may be used for reference in the same intra-period. Embodiments provide corresponding methods permitting resuming coding at a specific picture (which need not be an Intra-frame) upon an encoding interruption without having to encode the preceding pictures again.
Such embodiments permit resuming the encoding process upon an interruption which is caused by a failure, for example, without having to encode the preceding pictures again to receive the information necessary to encode a picture which is not a picture that can be encoded independent of any other pictures (Intra-frame or I frame). Since decoding an encoded picture to receive the information necessary is generally less time consuming when compared to encoding the picture, such embodiments allow for resuming encoding in a reduced amount of time.
Embodiments of the present invention provide for a computer program having a program code for performing, when running on a computer, any of the methods of the present invention.
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
Embodiments of the present disclosure relate to approaches for improving the performance of video encoders and/or video decoders.
Algorithmic approaches may be used to improve the performance of video encoders and/or decoders.
A possible approach of video encoder optimization is to prune the encoding search space, i.e. to implement shortcuts, which reduce the number of evaluated coding options. Following the concept of rate-distortion optimized encoding (RDO), different encoding options are evaluated and finally the option with the smallest rate-distortion cost D+λ R is chosen. In current state of the art video encoder implementations numerous RDO based decisions have to be performed, which may include further RDO based sub-decisions, leading to a complex and comprehensive decision tree.
Typical RDO based decisions include a mode search, where all modes of a particular partition size, e.g. intra and inter modes, are tested and an evaluation of all available split options is performed. Especially exploring multiple split options will cause an exponential increase of evaluated coding options. If, due to constrained computational resources, an exhaustive search of all these options is not feasible, fast termination rules could be applied to reduce the number of tested options. Fast termination rules try to identify and avoid testing of improbable coding options by deriving early termination rules based on the analysis of the encoded content and/or the results of already tested encoding modes. These termination decisions may affect single encoding modes or groups of modes like intra or inter coding at a particular partition size. Early termination decisions can also lead to pruning split decision trees, if the testing is not likely to contribute coding gains. Such a pruning typically leads to a significant timesaving.
Content independent early termination rules may be used. Splitting trees are used in block-based video coding standards, such as HEVC (High Efficiency Video Coding), and the overall coding costs of a block depend on the tree signaling of splits.
For example, a picture partitioned in coding blocks and sub-blocks is shown in
Splits that are more expensive in signaling may be limited by assigning them separate tree depths whose limits are set a priori. Instead of spanning a full decision tree expensive branches may be pruned off at smaller depths and lead to a faster determination of the searching, if it is expected not to have a large depth, as it is the case for higher temporal layers in the random access configuration of HEVC. These temporal layer dependent depths can also be signalized and then be used on the decoder side to optimize the overall signaling.
Content dependent termination rules may also be used. In the content dependent case, additional information from the content may be gathered in order to deduce a possibility that a certain split should be tested. In this method, directional gradients may be computed in order to find edges inside the block so that the partitioning can localize the objects of interest. For example, if the sum of differences along the vertical direction is greater than the sum of differences along the horizontal direction, then horizontal information is expected and hence a split in vertical direction is not performed. On the other hand, if horizontal information is expected, then the differences on each side are compared in order to localize the edge information and then a split according to this information is performed. In order not to make too many false implications this rule may only be active for larger blocks as the edges of objects differ more significantly from textured information. A further threshold may be introduced to reduce false implications. A straightforward extension of this may be to consider higher order differences in order to obtain higher accuracy. Methods that are more sophisticated could involve additional transforms, e.g. directional transforms or learned transforms, which give rise to more geometrical information that can be used to deduce a split decision.
Accordingly, embodiments provide a video encoder for encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, wherein the video encoder is configured to decide on a recursive partitioning of coding blocks into smaller sub-blocks by evaluating different encoding split options including a first split option splitting a coding block or sub-block in a first direction, wherein the video encoder is configured to: determine a first similarity of pixels of a block or sub-block along the first direction; and if the first similarity fulfils a predetermined condition, not evaluate the first split option or defer evaluating the first split option to a subsequent decision cycle.
Thus, embodiments permit usage of a content dependent measure or metric, i.e., the similarity between pixels in the respective block or sub-block, to deduce whether a specific split option should be evaluated (tested) or not. Generally, a similarity measure may be used that is in inverse proportion to the similarity, such as the sum of differences or the sum of gradients between pixels in the respective block or sub-block in the respective direction. Generally, similarity means mutual similarity between respective pixels in the block or sub-block. Generally, the similarity may be determined based on the brightness values (Luma) of the pixels. Chroma values of the pixels may also be considered in determining the similarity.
In embodiments, the video encoder is configured to determine the similarity based on the differences of locally neighboring pixels in the block or sub-block.
In embodiments, the predetermined condition may be that the similarity is less than a similarity threshold (or that the similarity measure indicating the similarity is larger than a corresponding threshold).
In embodiments, the video encoder is configured to determine a second similarity of pixels of the block or sub-block along a second direction different from the first direction, wherein the predetermined condition is fulfilled if the first and second similarity comprise a predetermined relationship. In embodiments, the predetermined relationship is fulfilled if the first similarity is less than the second similarity or if the first similarity is less than the second similarity by at least a specific amount. In embodiments, the specific amount may be a fixed amount or may be a specific percentage of a value indicating the first similarity, such as 20%.
In embodiments, the first and second directions are perpendicular to each other. In embodiments, the first direction is the horizontal direction and the second direction is the vertical direction. In other embodiments, the first direction is the vertical direction and the second direction is the horizontal direction. The terms “horizontal” and “vertical” relate to a lower boundary of the picture, such as picture 100 in
In embodiments, the first split option is an asymmetric horizontal split, wherein the video encoder is configured to not evaluate the first split option or defer evaluating the first split option to a subsequent decision cycle if the similarity along the vertical direction is less than the similarity in the horizontal direction and the similarity in the vertical direction in the halve of the block or sub-block, in which the split would be located, is less than the similarity in the vertical direction in the halve of the block or sub-block, in which the split would not be located. In embodiments, the first split option is an asymmetric vertical split, wherein the video encoder is configured to not evaluate the first split option or defer evaluating the first split option to a subsequent decision cycle if the similarity along the horizontal direction is less than the similarity in the vertical direction and the similarity in the horizontal direction in the halve of the block or sub-block, in which the split would be located, is less than the similarity in the horizontal direction in the halve of the block or sub-block, in which the split would not be located.
In embodiments, the video encoder is configured not to evaluate splits in the first direction if the sum of absolute differences between the pixels along the first direction is greater than the weighted sum of absolute differences between the pixels in the vertical direction.
In embodiments, a directional split is not tested if a similarity measure along the split direction is greater, so to say less similar, than the weighted similarity measure in a perpendicular direction. In embodiments, horizontal splits are not tested if the similarity measure along the horizontal direction are greater than the weighted similarity measure in vertical direction. In embodiments, vertical splits are not tested if the similarity measure along the vertical direction is greater than the normalized similarity measure in horizontal direction. In embodiments, an asymmetric horizontal split is not tested if the similarity metric along the vertical direction is greater than the weighted similarity metric in horizontal direction and the similarity measure in vertical direction in the area that covers the split is greater than the weighted similarity measure in vertical direction of the subblocks that do not cover the split. In embodiments, an asymmetric vertical split is not tested if the similarity metric along the horizontal direction is greater than the weighted similarity measure in vertical direction and the similarity measure in horizontal direction that cover the split is greater than the weighted similarity measure in horizontal directions of the subblocks that do not cover the split. In embodiments, horizontal splits are not tested if the sum of absolute differences along the horizontal direction are greater than the weighted sum of absolute differences in vertical direction. In embodiments, vertical splits are not tested if the sum of absolute differences along the vertical direction are greater than the weighted sum of absolute differences in horizontal direction. In embodiments, an asymmetric horizontal split is not tested if the sum of the gradients along the vertical direction is greater than the weighted sum of gradients in horizontal direction and the sum of vertical gradients that cover the split is greater than the weighted sum of gradients in vertical directions of the subblocks that do not cover the split. In embodiments, an asymmetric vertical split is not tested if the sum of the gradients along the horizontal direction is greater than the weighted sum of gradients in vertical direction and the sum of horizontal gradients that cover the split is greater than the weighted sum of gradients in horizontal directions of the subblocks that do not cover the split. In embodiments, weighting of a similarity or a similarity measure takes place dependent on, but is not limited to, the size of the block the similarity or similarity measure is associated with. In embodiments, weighting may alternatively or additionally depend on the split direction associated with the similarity of similarity measure.
In such embodiments of the invention, in which specific split options are not evaluated, generally, an arbitrarily shaped area is divided into n disjoint parts. Then finite differences in m different orientations may be computed, resulting in m*n values. These values can be weighted, for instance, according to the area. Then split modes are neglected, i.e. are not evaluated, if the split direction is within a certain angle range of the gradient direction that is most dominant. All other splits are not neglected, i.e. are evaluated.
According to particular embodiments, the angle range may be set to 0.
An example of applying content dependent termination rules is now described referring to
Then gradients in horizontal and vertical directions are computed based on first order finite differences. The gradients are then summed up in absolute values for the four quadrants shown in
If the sum of the four horizontal gradients is larger than the sum of the four vertical gradients then symmetric and/or asymmetric splits in horizontal direction are not tested.
If the sum of the horizontal gradients in blocks P2 and P3 are greater than the sum of horizontal gradients in P1 and P4, then vertical splits that are located in quadrants P1 and P4 are not tested (evaluated), provided the sum of all horizontal gradients in P1, P2, P3, and P4 are greater than the sum of vertical gradients in P1, P2, P3, and P4. If the sum of the horizontal gradients in blocks P1 and P4 is greater than the sum of horizontal gradients in P2 and P3, then vertical splits that are located in quadrants P2 and P3 are not tested, provided the sum of all horizontal gradients are greater than the sum of vertical gradients.
The same speed up technique may be used in orthogonal directions with the vertical gradients. For the localized gradients the blocks P1 and P2 are grouped and P3 and P4 are grouped, respectively.
For example, in a step 1, four gradients in horizontal direction h1, h2, h3, h4 and four gradients in vertical direction v1, v2, v3, v4 may be computed. In a step 2, the sum of the four gradients in horizontal direction may be calculated and the sum of the gradients in vertical direction may be calculated. If h1+h2+h3+h4>v1+v2+v3+v4, horizontal split options are not evaluated. In such case, only vertical spit options would be evaluated, i.e. are in contest. Thereupon, in a step 3, it may be checked whether h2+h3<h1+h4. If so, an asymmetrical left vertical split is forbidden.
Part b) of
Examples of how differences or transforms to derive a similarity measure may be calculated are now described. The calculations may be done based on the respective brightness values of the pixels.
Given a block B with width w and height h horizontal differences h1, h2, h3 and h4 may be computed as follows:
Vertical differences may be computed as follows:
Higher order differences may also be used to determine the similarity:
In other embodiments, transformed errors is using a transformation T, such as DCT (discrete cosine transform) or others, may be used to compute the similarity or the similarity measure:
wherein C is the transformed block of B by T, that is C=TB. In other embodiments non-linear transforms may be used.
In other embodiments, higher order differences may be used, for example:
In the state of the art hybrid video codecs, like H.265/HEVC, the partitioning search is based on a top-down approach, in which a currently best candidate is compared with best results for single or multiple split options. To obtain the best result for a split option, an exhaustive recursive search has to be done to evaluate all options, inclusive sub-splits, for that split. This causes the partitioning RD-search to create a search domain, the size of which increases exponentially with the number of available depths, and splits at one level. To reduce this search domain, in examples, a shortcut allowing for fast split screening may be introduced. In this method, a step may be introduced before the currently best candidate is compared to the best splitting results. Before the best split result is evaluated for a specific split with a top-down RD-search, a fast coarse search may be performed providing a result for this split. This result may than be compared with the currently best candidate. Only if the coarse result proves to be better or only worse by less than a predefined margin or ratio, a deep top-down RD-search for the best results for a given split is performed. The coarse search might be performed by splitting the current coding block and finding the best result for each sub-partition without further splitting (thus, searching within all possibilities but further splits). If the deep RD-search is performed based on the results of the coarse search, the mode constellations examined during the coarse search should be excluded. The domain searched if a deep RD-search is performed might still be restricted by this or other shortcuts and does not need to a full recursive top-down search.
Thus, embodiments provide a video encoder for encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, wherein the video encoder is configured to recursively partition coding blocks into smaller sub-blocks, wherein the video encoder is configured to: for a particular coding block or sub-block, evaluate all or a subset of non-partitioning encoder modes, probe a specific partitioning encoder mode by evaluating all or a subset of non-partitioning encoder modes for each of the sub-partitions of the specific partitioning encoder mode, but excluding further partitioning of the specific partitioning encoder mode, wherein the specific partitioning encoder mode is excluded from a complete evaluation if the RD-cost of the probed specific partitioning encoder mode exceeds a threshold.
In embodiments, the threshold is set to lowest RD-Cost of the evaluated non-partitioning encoder modes of the particular coding block or sub-block.
In such embodiments, the RD cost of the probed specific partitioning encoder mode may be determined from RD cost associated with all or the subset of non-partitioning encoder modes of each of the sub-partitions of the specific partitioning encoder mode. In embodiments, the RD cost of the probed specific partitioning encoder mode may be the lowest RD cost of these. In embodiments, the RD cost of the probed specific partitioning encoder mode may be determined by determining, for each of the sub-partitions, the lowest RD cost and by summing these lowest RD costs.
Thus, embodiments permit for excluding encoding mode options which turned out to be not appropriate from a full evaluation so that the time that may be used for encoding the pictures may be reduced.
In examples, a multi pass encoding may be performed. Under the assumption that the split structure, or encoding decisions are primarily influenced by the local content to be encoded and by the usability of available spatial and temporal prediction candidates, in many cases, some unlikely encoding mode candidates or unlikely split branches can be identified and could be excluded from the mode search.
To simplify the analysis step for a coding block (CTU) with a large number of possible decisions, a generic multi pass approach may be introduced. The encoder may be run in a first pass with a strongly reduced set of encoding modes and/or split branches, whereas in the succeeding runs more and more of these initially imposed restrictions may be removed but other restrictions may be added according to results gathered in the previous runs.
The results gathered from a run (pass) are stored in a buffer individually for each run. The stored information includes but are not limited to the mode cost and which encoding mode was tested. A unique identifier is used to store the information, gathering all the entries that are assigned to the identifier.
During a succeeding run, the mode control can look up, with the current identifier, information about the decisions taken in the preceding run. The mode control decisions in the current pass can be manipulated according to conclusion drawn from results of the preceding passes. This can lead to excluding or restricting one or more encoding modes or split-branches, or groups of these with similar properties, from investigation, or emphasizing promising encoding modes or split branches by relaxing mode specific restrictions.
In an embodiment, the multi-pass approach may be applied to each coding block (CTU) individually. For a decision node, evaluating encoding options for a particular partition, a unique identifier is formed from the position, size and split history of this partition. The mode costs generated in a preceding run are used to classify encoding modes and split-branches, by determining the minimal costs of the node generated in the first run. The minimal cost may be scaled by a parameterizable scaling factor to obtain a threshold. All potential encoder modes and split candidates may be checked prior to testing. If the cost of a mode, equal or similar to the current mode, exceeds the threshold, the particular mode is excluded from the mode search. In this context, the phrase ‘similar’ is defined as true for modes that share the same category, e.g. split/non-split, intra/inter, intra angular predicted/intra non-angular-predicted.
Thus, embodiments provide a video encoder for encoding pictures of a video into a data stream, the picture being partitioned into an array of coding blocks, wherein the video encoder is configured to recursively partition coding blocks into smaller sub-blocks, wherein the video encoder is configured to: perform a first recursive run, in which a sub-set of encoder mode options less than all available encoder mode options are evaluated for a particular coding block or sub-block and to store mode decision information associated with each of the sub-set of available encoder mode options; perform a second run, in which encoder mode options are evaluated for the particular block or sub-block, wherein encoder mode options determined to fulfill a specific exclude criterion are excluded from the second run, wherein the video encoder is configured to determine whether an encoder mode option fulfills the specific exclude criterion using the stored mode decision information.
In embodiments, a subset of the stored mode decision information is used, which is restricted to stem from an encoder mode option evaluated in the first run that is a collocated partition of the same partitioning size as the particular block or sub-block in the second run. In embodiments, a subset of the stored mode decision information is used, which is restricted to stem from an encoder mode option evaluated in the first run that is a collocated partition of the same partitioning size and the same split history as the particular block or sub-block in the second run.
In embodiments, the mode decision information is the rate distortion cost, RD cost.
In embodiments, the specific exclude criterion for a mode option is that the mode decision information or a weighted version of the mode decision information associated with the encoder mode option exceeds a threshold. Again, weighting of the mode decision information permits privileging some mode encoding options when compared to others.
In embodiments, the specific exclude criterion for a partitioning mode option is that the partitioning mode option to be tested in the second run has not been evaluated in the first run and that the mode decision information associated with a partitioning mode option having at least one identical property exceeds a threshold. The identical property may, for example, be the same split direction, same number of resulting sub-blocks or whether the option concerns a split at all. Thus, a similar partitioning mode option may be excluded from the second run. For example, a vertical split option partitioning a block or sub-block into two sub-blocks of the same size may be regarded as being similar to a vertical split option partitioning the block or sub-block into two blocks of different sizes. Thus, split options having one, two or more identical properties, but not all, may be regarded as being similar, such as the same split direction and the same number of resulting sub-blocks.
In a special case where only one split mode is tested in the first run, e.g. a quad-split with no advantageous split direction, the identical property might be that the tested partitioning mode splits the block anyhow.
In embodiments, the mode decision information is the RD-cost. As used herein, the term RD-cost also includes RD-cost scaled by a multiparametric function, including a multiplication of the value thereof with a scaling factor as well as an additive offset. For example, the RD-cost may be determined as a scaled value as follows: ScaledValue=ScalingFactor×Value+ScalingOffset. Value is the original RD-cost, ScalingFactor is a scaling factor and ScalingOffset is an additive offset. Scaling the RD cost permits weighting different encoding mode options differently so that some options, such as options concerning larger blocks or sub-blocks may be privileged
In embodiments, the mode decision information is the RD-cost and the threshold is set to the RD-cost of an encoding mode option of the first run having the lowest RD-cost. In embodiments, the mode decision information is the RD-cost and wherein the threshold is set to the RD-cost of a partitioning encoding mode option of the first run having the lowest RD-cost. In embodiments, the mode decision information is the RD-cost and wherein the threshold is set to the RD-cost of a partitioning encoding mode option of the first run having the same split direction, which has the lowest RD-cost. In embodiments, a partitioning mode option that has not been evaluated in the first run is not excluded from the second run. Thus, embodiments allow for excluding encoding mode option in a second or subsequent run in an appropriate manner.
In embodiments, the specific exclude criterion for a non-partitioning mode option is that the non-partitioning mode option to be tested in the second run has not been evaluated in the first run and that the mode decision information associated with a non-partitioning mode option having at least one identical property exceeds a threshold. The corresponding property may concern the kind of the mode, such as whether the mode is an intra predicted mode, whether the mode is a skipped mode, and whether the mode is a motion compensated mode. Thus, encoding mode options having similar properties may be excluded from the second run.
In embodiments, the mode decision information is the RD-cost, and, for a non-partitioning encoding mode option, the threshold is set to the RD-cost of an encoding mode option of the first run having the lowest RD-cost. In embodiments, the mode decision information is the RD-cost, and, for a non-partitioning encoding mode option, the threshold is set to the RD-cost of a non-partitioning mode having the lowest RD-cost. In embodiments, a non-partitioning mode option that has not been evaluated in the first run is not excluded from the second run. Thus, embodiments allow for excluding encoding mode option in a second or subsequent run in an appropriate manner.
The above approaches for excluding encoding mode options from a second run may also be applied to any subsequent run of the recursive execution, wherein a respective subsequent run corresponds to the second run and a respective preceding run corresponds to the first run.
A powerful way of decreasing program execution time is parallel processing. Support for parallel processing is an inherent feature of today's custom of the shelf computer hardware and what's more, these parallel capabilities will further increase with future CPU (central processing unit) generations. Apart from a parallel software solution executed on CPUs, the use of GPUs (graphics processing units) provides additional resources for massive parallel processing.
To benefit from parallel execution, it is advantageous that the program under optimization provides well-structured data sections and a clear program flow. Applying parallel execution capabilities to a sequential program can cause errors due to concurrent data accesses that are usually hard to detect.
Another issue that comes with parallel processing is deterministic behavior. Most programs may be used to operate deterministic, so the outcome is identical when the input is the same. When executing tasks in parallel, deterministic behavior is not necessarily required —but desired. For example, a video decoder should always reconstruct the input bit stream in a deterministic way. Opposite, a video encoder might be allowed to operate non-deterministically under the constraint that the resulting bit stream is correct and the performance is not degraded. However, non-deterministic behavior will hamper the program verification and might hide undetected internal program issues.
To preserve deterministic behavior, there are some requirements to operate parallel program sections. Such sections may have no side effects on other sections. If there are dependencies that cannot be neglected, then these dependencies have to be modeled by synchronization of the program sections. However, correct program behavior excludes concurrent data access. If inter-process communication may be useful, it has to be secured using thread safe techniques.
The present disclosure discusses different parallel processing approaches that can be applied to a video codec. The approaches are discussed for video encoder, but are not limited to it. The proposed different approaches are outlined one by one but are not mutually exclusive and can also be applied in combination, where the speed up rates ideally multiply.
The video decoder 70 comprises processing means 80a, 80b, 82, 84, 86, which are configured to perform any processes necessary to decode encoded data stream 74 and to generate the series of pictures from the encoded data stream 74. Some of the processing means, such as processing means 82, 84, and 86 are configured to perform tasks in parallel. In embodiments, processing means 82, 84 and 86 may be formed by threads in a computer environment. In other embodiments, processing means 82, 84, 86 may be formed by separate hardware units permitting parallel execution of tasks.
In embodiments, the video encoders and the video decoders described herein may be implemented in hardware using analog and/or digital circuits, in software, through the execution of instructions by one or more general purpose or special-purpose processors, or as a combination of hardware and software. In embodiments, the video encoders and the video decoders may be implemented as a computer system or another processing system. The computer system or processing system may be programmed or configured to provide for the functionality described herein. The computer/processing system may include one or more processors, like a special purpose or a general purpose digital signal processor. The processor(s) may be connected to a communication infrastructure, like a bus or a network. The computer/processing system may include a main memory, e.g., a random access memory (RAM), and a secondary memory, e.g., a hard disk drive and/or a removable storage drive. The secondary memory may allow programs or other instructions to be loaded into the computer/processor system. The computer/processor system may further include a communications interface to allow software and data to be transferred between computer/processor system and external devices. The communication may be in the from electronic, electromagnetic, optical, or other signals capable of being handled by a communications interface. The communication may use a wire or a cable, fiber optics, a phone line, a cellular phone link, an RF link and other communications channels.
The coding blocks preceding coding block 112 in the processing sequence are available for reference and the coding blocks following coding block 112 in the processing sequence are not available for reference. Coding blocks which are actually processed, coding blocks, which are available for reference, and coding blocks which are not available for reference are depicted differently.
Processing a coding block may involve reference to other coding blocks, and such other coding blocks are sometimes herein called reference coding blocks. Reference to reference coding blocks may be that may be used for use as source for spatial prediction or for use as source for context derivation in case of context adaptive entropy coding, for example.
As indicated by arrows in the lower left part of
Wave front parallel processing (WPP) is a technique to parallelize video encoder and decoder applications. WPP has been adopted in the current state of the art HEVC standard [1]. If WPP is enabled, specialized features are used. The operational granularity is a CTU. Data dependencies between neighboring CTUs are obvious and undesired data races can be avoided by guaranteeing a minimum CTU offset of two between consecutive CTU-lines as shown by the currently processed CTUs 112 in the upper right part of
New tools incorporated in future standards may introduce additional dependencies and therefore may use an increased CTU-offset between consecutive CTU-lines in the WPP case. This can be compensated by either restricting the tool or disabling the tool whenever the outcome might be affected by race conditions. As shown by the arrows in the lower parts of
A first approach is based on a so-called lockstep execution and is shown in the right part of
Dynamic Approach with Local Synchronization Objects
A second approach for synchronizing parallel processed coding blocks (CTUs) is a dynamic approach with local synchronization objects, i.e., using explicit synchronization. Each CTU-line is processed by one task. Each task may be performed by a thread. The left hand part of
In this variant, two or more tasks are running independently, but are synchronized to prevent overtaking of the superior task by the dependent one, ensuring the useful minimum CTU-offset. For synchronization, synchronization objects, such as flags, are used. For synchronization, either a single synchronization object is shared between the tasks and each CTU or each CTU-line is assigned to shared synchronization object.
Before processing a CTU, the task has to test whether all dependencies are resolved. Therefor it accesses a thread safe variable. If all dependencies are resolved then the task can process the CTU, otherwise, if any of the useful dependencies remains unresolved, the task has to suspend processing until all resources are available. After processing a CTU, the corresponding synchronization object is modified to indicate that the CTU has been processed. Under the assumption that each CTU-line is processed by one task, and they only depend on the previous CTU of the same CTU-line, with an always solve reference, and on CTUs that belong to the CTU-line above, then one task just has to check whether the CTU according to the fixed CTU-offset of the CTU-line above has been processed already.
For the synchronization, it is not relevant how the state of the last processed CTU is derived. The information can be stored in a thread safe variable, that is only shared between the two tasks, or it could be stored for each CTU-line and, thus, a task aware what CTU-line is currently processed can directly access the information. Alternatively, the information could be stored in a thread safe variable assigned to each CTU, thus a thread aware of its CTU position could directly access the rightmost reference CTU-variable to get the information. However, the task processing the first CTU-line is not affected by synchronization, and does not have to wait for unresolved references. When the second task is started it checks the synchronization object and if this is not blocked the task can process the next CTU. When the CTU is done, the task modifies the corresponding synchronization object to indicate the CTU is done, before checking the next CTUs synchronization object. Once a task is blocked due to an unresolved reference indicated by a blocked synchronization object, it has to wait until the reference is solved and marked available by unblocking the synchronization object. The benefit of this method is a higher throughput due to more independent processing with minimal synchronization overhead. This approach also benefits from data locality.
Embodiments of the present invention provide an approach to improve performance when processing CTU lines in parallel by different threads (tasks). In particular, in case task resources and, therefore, threads, are limited, the throughput of CTUs can be increased according to embodiments of the invention. To be more specific, in environments as explained above with respect to
CTUs 112, 122 are currently processed by the first and second thread. The third thread processing CTU line 13 is blocked since the second thread has not yet finished processing CTU 122. Thus, the third thread may not proceed with processing CTU 124 but has to wait until the second thread has finished processing of CTU 122 (since CTU 122 is a reference CTU for CTU 124). Thus, the third thread is idle.
In order to avoid such idle times, in embodiments to the invention, the third thread interrupts processing line 13 and jumps to the next line not yet or not currently processed by another thread and starts processing a CTU in this line, which is available for processing (since all reference CTUs for this CTU have already been processed). In
Upon having processed CTU 126, the third thread may check whether CTU 124 is still blocked and may jump back to CTU 124 if it is no longer blocked. Otherwise, it may proceed with processing the next CTU in line 14 as indicated by an arrow 132 in
A master thread may be provided, which may control the allocation of tasks to the threads. The master thread may control how the switching of threads to another line is to be performed upon the specific event. The master thread may be configured to control the allocation of tasks so as to keep idle times as low as possible.
Thus, in embodiments of invention, the processing schemes explained above with referring to
In embodiments of the invention, the specific event upon which processing one CTU line is interrupted may be that processing this line reaches a specific CTU.
While there is an offset between jumps 140 and 142, processing a first CTU in a line may be more time consuming than processing subsequent CTUs. Thus, if the first thread would have to process the first CTU in the fifth line, the second thread might be blocked from processing the first CTU in the sixth line due to the fact that the first thread did not yet finish processing the first CTU in the fifth line. Thus, the second thread would be idle. This can be avoided by embodiments of the invention since the first thread has been processed in advance by the fourth thread.
Thus, in embodiments of the invention, the tasks that operate on the last regular CTU-line are interrupted at a specific position if the succeeding CTU line has not been started. The computational resources of the interrupted tasks are handed over to the task of the line below the stopped one. This helps to prevent potential blocking of tasks at the beginning of the next CTU-lines.
The processing schemes described referring to
Embodiments relate to approaches using parallel processing within coding blocks, such as CTUs. Besides parallel encoding of CTUs as used with wave front parallel processing, parallel encoding within a CTU may also be a promising option for accelerating the overall execution time. Given a certain partition size several encoding modes are evaluated before a final mode decision is taken. Because there are no data dependencies between these encoding modes, they can be tested in parallel. Without restriction, this approach can be extended to testing of available split modes. Current state of the art [3] implementations of future video codes utilize more than one split mode. With further enhancements, which introduce even more split options [5], [6] the potential of parallelization is increasing.
The main advantage of split parallelization versus mode parallelization is the longer execution time of splits when compared to other modes. The more splits are available, the more this statement becomes true. This is caused by the exponential growth of the search space with respect to the number of splits, while the search space for non-split modes is unchanged. Also, longer execution times cause fewer synchronization points reducing multi-threading overhead.
In [3], [5], and [6] partitioning methods are described which allow to recursively partition a coding block or macro-block (CTU) into smaller block for further encoding. Other than in the current standard H.265/HEVC, all of these methods allow a multitude of splits instead of only allowing the quad-split. This flexible partitioning enables optimal representation of spatial and temporal dependencies in further coding. This multitude of splits can process in parallel. In a possible implementation, a thread pool and separate encoding stacks are prepared for each thread. Encoding may start as usual. After starting encoding of a CTU, different modes will be probed at each partitioning level, some of the further splits. At well-defined parallelization points, instead of testing all modes sequentially, the modes will be grouped into disjunct sub-sets. The sub-sets will then be probed in parallel, producing a best-mode for each of the sub-sets. All modes from a specific sub-set need to be tested independently of any modes from different sub-sets. Dependencies for modes within a sub-set are allowed. The sub-sets can be executed on an arbitrary number of threads, at least one. Increasing the number of threads beyond the number of subsets will not provide further improvements. After all of the sub-sets have been executed, best mode has to be chosen from the best candidates calculated for each sub-set. After the parallel execution of mode testing at a specific well defined parallelization point, sequential execution is resumed. When parallelizing splits at a parallelization point, all non-split modes can be combined to a separate sub-set and executed in parallel as any other sub-set. Nesting of parallelization points can be applied, as long as not all threads of the thread-pool are busy. The disjunct sub-sets can be different, depending on mode availability. Ensuring the parallelization points as well as modes subdivision into sub-sets are well-defined and independent of the number of available threads may ensure the encoder to always produce the same results.
Exemplary subsets for [3] may be: all non-split modes, QT-split, horizontal BT-split, and vertical BT-split. All elements are subjected to availability within the specific block.
Exemplary subsets for [5] may be: all non-split modes, QT-split, horizontal MTT-bi-split, vertical MTT-bi-split, horizontal MTT-tri-split, vertical MTT-tri-split. All elements are subjected to availability within the specific block. MTT stands for multi-type-tree. In addition to quad-tree and binary-tree, triple-tree partitioning may be employed in multi-type-tree structures. In a Quad-Tree-Binary-Tree structure, a CTU is firstly partitioned by a quad-tree structure and the quad-tree leaf nodes are further partitioned by a binary-tree structure. In a multi-type tree structure, horizontal and vertical center-side triple-trees are introduced, in which a block or sub-block is partitioned into three sub-blocks, wherein the size of the center sub-block may be twice the size of the side sub-blocks.
Other exemplary subsets if only split ratios ½, ¼, ¾, ⅓, ⅔ are allowed may be: all non-split modes, perpendicular ½ split, parallel ½ split, perpendicular and parallel ¼ split, perpendicular and parallel ¾ split, perpendicular and parallel ⅓ split, perpendicular and parallel ⅔ split. All elements are subjected to availability within the specific block.
When considering parallelization within coding blocks, load balancing should be a concern. For successful parallelization all threads running in parallel should last similarly long. As the overall execution time is defined by the slowest thread, if other threads are significantly faster, resources are wasted. To accommodate this obstacle, load-balancing techniques may be used. In the state of the art [3] implementation of a future video codec, two tree types are intermixed. At the CTU-level a quad-tree (QT) is started. In a quad-tree, a square block can be recursively split into four smaller squares. In [3], the quad-tree has no restrictions as long as further splitting results in valid codable blocks of not smaller than a pre-defined size. At each leaf of the quad-tree, a binary tree (BT) is started. In a BT, a rectangular block is recursively split into two rectangular sub-blocks of equal size. The split can be performed along the vertical or horizontal axis. In [3], a BT is restricted to a specific depth, thus limiting the search space. The leafs of a BT are coded without further subdivision. As BT is embedded into QT, BT extends the search space of QT, creating a much larger space than this of BT only. For this reason, QT splits usually take longer to test than BT splits. To take this fact into account during load balancing of split-parallelization, the parallelization is not applied at the CTU level, as the QT search space is much larger that BT search space. Optimally, the parallelization should be applied when the search space of the QT split roughly equals the search spaces of BT-splits. The same applies for [5], if the QT is balanced against the multi-type-tree rather than the BT. For an implementation of [5], it was found that parallelization at block size 16×16 is optimal.
In other embodiments, the parallelization switch point can be adaptively set across frames within the same T-layer, frames with the same QP (quantization parameter), all frames in the coding order. If a BT split or MTT split is started in a non-parallel segment, it is assumed that all further BT-splits (or MTT-splits) have similarly large search spaces. For this reason, those splits can be executed in parallel.
Thus, embodiments of the present invention provide for parallelization within coding blocks, such as CTUs, wherein such parallelization takes place when a specific criterion is met. As described above, the criterion may be that the block has a specific block size or less, wherein the specific block size is less than the coding block size. Another criterion may be that a quad split is not available for the block or sub-block.
Thus, according to embodiments of the invention, a video encoder, such as the encoder shown in
As shown in
Analogue to different split options, current state of the art of future video codecs [3] allows many non-split modes to be applied to a leaf of the partitioning tree. Sophisticated prediction and transformation modes increase the complexity. Thus, in embodiments, those modes may be executed in parallel. In addition or as an alternative, embodiments may create disjunct sub-sets of all non-split modes within a specific block, which are than executed in parallel, similar to split parallelization. After the parallelization has finished and a best mode for non-split has been established, split-modes can be executed sequentially. Alternatively, testing of split modes can be executed in parallel as described above, without creating a designated sub-set for non-split modes (which already were tested for this specific block). In [3], the tested non-split modes produce many interdependencies, which need to be taken into account when defining the sub-sets. For example, some modes might be only refined results of other modes—for those modes it may be crucial that they are executed within the same sub-set (as the can be no dependencies between the sub-sets). Exemplary sub-sets for non-splits modes for [3] may be: Intra-modes, Merge- and Skip-Modes (all Inter-Modes for which no MVD (multivalued dependency) will be signaled), Motion-Search modes (non-merge) (all Inter-Modes for which a MVD will be signaled). All modes are subjected to availability constraints for a particular block.
In addition to parallelization in processing coding blocks, such as CTUs, and within coding blocks, pictures may be encoded in parallel.
One of the commonly used configurations in video coding is a “random access” coding mode, where a hierarchical picture formation is packed into a group of pictures (GOP). The structural delay, caused by some dependencies between pictures, allows picture parallel processing inside GOP as well as between GOPs.
The theoretical limit for the acceleration through GOP-level frame parallel coding is determined by the GOP-size and processing time for a single picture. Due to the hierarchical structure, the larger the GOP-size the more temporal levels are available and hence, the more CPU-cores can be utilized. Dependent on the temporal layer, processing time for a single picture can have significant variations. In a tested encoder configuration, temporal level-0 pictures showed about 30-50% of total encoding time in sequential mode. Due to the fact, that the lowest temporal level pictures can only be processed sequentially, the rest of the processing time should be optimally distributed over the remaining pictures processed in parallel. So, the temporal level non-zero pictures of one GOP should be processed as fast as or even faster than the temporal level zero picture in the following GOP.
For an optimal parallel processing, a particular scheduling algorithm with prioritization of temporal level-zero pictures was implemented. A brief description of the algorithm is given below. An example of the encoding process utilizes two picture lists: Input-List and Processing-List. Pictures are coming in in encoding (decoding) order. All incoming pictures are put into Input-List. The algorithm goes through Input-List, taking pictures from Input-List in a particular processing order and puts them into Processing-List. Then the pictures from Processing-List are submitted to the multi-threaded framework in a First-In-First-Out (FIFO) order.
The algorithm is:
In another embodiment, when encoding more than one intra-period GOP with multiple random access points, the temporal-level zero pictures including the intra picture (I-Slices) can be processed parallel to temporal level zero pictures of other intra-period GOP. For example, in
In addition to parallelization in processing pictures, coded bitstreams may be reused. Originally with the intention to introduce helpful debugging capabilities, a decoder may be embedded into the encoder. Generally, a decoder is a magnitude faster with decoding a picture than it takes an encoder to encode this picture. In a debugging case, a faulty bit-stream is in most of the cases available. Beside the targeted debugging option, the decoder may be a very useful feature in other applications. The decoder parses a bit-stream up to some parameterized picture and reconstructs the pictures, all symbols and reconstructed data are copied to the encoder, the encoder does not perform a mode decision but writes the bit-stream. When the decoder stops decoding, the encoder may take over and continue using the common encoding process.
The information concerning a picture, which is derived by decoding the picture and which is supplied to the encoder to permit the encoder to encode non-RAP-pictures or to resume encoding may include the following. This information may include all information available for a decoded picture. This information includes reconstructed samples, decoded symbols, including temporal prediction information, (like motion vectors, reference picture indices, motion vector predictor indices), intra prediction information (like intra prediction modes, internal statistics of the prediction chain), mode information (like skip, intra, inter), and pre- and post-processing filter parameters. The information may also incorporate intermediate data (e.g. residual signals, statistical information, partitioning statistics) generated by the decoder during the decoding or filtering process. The encoder may use additional evaluation of the information received from the decoder in order to derive the information, that might be used for encoder optimization, to achieve deterministic behavior or to generate bit-equal bitstreams compared to an encoder only solution.
Embodiments relate to a parallel encoding of GOPs separated by I pictures (I slices). Video streams used in broadcast, streaming, etc. are equipped with random access points (RAP) that allow video decoders to tune into the video at predefined entry points. These RAPs can be used to fast forward or fast backward videos. A common approach to speed up the encoding of an entire video is to split it into chunks that can be encoded individually, usually located between these random access points. RAP-pictures can be encoded and decoded without reference to any other picture. In contrast thereto, non-RAP-pictures can generally not be encoded or decoded without reference to other pictures, such as the RAP-pictures preceding and following a group on non-RAP-pictures.
Embodiments of the present invention permit avoiding such redundancy. As shown in
The encoding units 302, 304, 306 encode the non-RAP-pictures of the respective sub-series of pictures. However, the encoding units 302, 304, 306 do not encode both RAP-pictures adjoining the non-RAP-pictures. Rather, in embodiments the encoding units 302, 304, 306 encode one of these RAP-pictures only or no RAP-picture at all. The encoding units may be formed by separate threads.
Rather, the encoding units 302, 304, 306 are configured to encode the non-RAP-pictures of a sub-series of pictures using information concerning at least one (or both) of the two RAP-pictures adjoining the non-RAP-pictures, which is obtained by decoding an encoded version of this RAP-picture, which has been encoded by another encoding unit. The encoding units may receive the information or may receive the version encoded by the other encoding unit (and may decode this version by an embedded encoder). This is indicated in
According to embodiments, the encoding units receive the encoded picture from the other encoder and decode the picture by an embedded decoder to obtain the necessary information. In other embodiments, a separate decoder may be provided from which the encoding units receive the necessary information.
As shown in
According to the embodiment of
Thus, in such embodiments, none of the RAP-pictures is encoded twice and, therefore, the overhead involved in such redundancy can be avoided.
Thus, in the embodiment of
In embodiments of the present invention, a decoder embedded in a video encoder may also be used for error robust encoding, especially if the encoder does not operate consequently deterministic.
When the interruption takes place, the decoder is in a position to provide the encoding unit with the information necessary to resume encoding directly at the image at which the interruption took place. Then the resulting data stream may be formed by combining the sub-stream encoded before the interruption and the sub-stream obtained upon resuming encoding after the interruption. Thus, the encoder is restarted after a termination caused by a potential fault. The bit-stream, written so far, that was generated in the faulty run is used with the embedded decoder to fast forward to the point of error, where the encoder continues the encoding process. Accordingly, the encoder need not encode the preceding pictures or at least the preceding pictures up to the preceding RAP-picture again.
In another embodiment, when using the embedded decoder to fast forward to the point of error, the decoder skips major parts of the parsing and reconstruction process for pictures that are not necessary for reference. This includes entire intra-periods, that precedes the erroneous intra-period, as well as non-reference-pictures, or pictures that are not that may be used for reference within the erroneous intra-period. This is shown in part C) of
Accordingly, embodiments of video encoders described herein may be adapted in order to deal with multiple input bit-streams as well as with the potential misleading picture-POCs of the decoded RAPs.
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a feature of a method or a feature of a method step. Analogously, aspects described in the context of a method also represent a description of a corresponding feature or functionality of a corresponding apparatus. Some or all of the methods may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In embodiments, one or more of the most important method steps may be executed by such an apparatus.
Depending on certain implementation nsisments, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to implement the video encoders and decoders described herein and/or to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to implement the devices described herein and/or to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are advantageously performed by any hardware apparatus.
The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.
The apparatus described herein, or any components of the apparatus described herein, may be implemented at least partially in hardware and/or in software.
The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.
The methods described herein, or any components of the apparatus described herein, may be performed at least partially by hardware and/or by software.
While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 18165065.6 | Mar 2018 | EP | regional |
This application is a continuation of copending International Application No. PCT/EP2019/057902, filed Mar. 28, 2019, which is incorporated herein by reference in its entirety, and additionally claims priority from European Application No. EP 18 165 065.6, filed Mar. 29, 2018, which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/EP2019/057902 | Mar 2019 | US |
| Child | 17033157 | US |
