Patent Application
20160037167
The current disclosure relates to decoding video bitstreams and in particular to improving the quality of decoded video bitstreams of varying quality.
Video can be encoded using different techniques. The encoded video may then be transmitted to a receiving device using a communication channel and the encoded video can be decoded and displayed. The encoding and decoding process may provide a tradeoff between complexity of encoding, complexity of decoding, quality of the decoded video, size of the encoded video, memory requirements for encoding and memory requirements for decoding. For example, the same video may be encoded to produce two different size encoded video files having the same visual quality, with the smaller sized video being more complex to encode and/or decode.
When streaming videos, for example over a network, videos may be encoded as individual video clips or segments that can each be independently decoded and stitched together into a single video. Each segment may be encoded a number of times to produce different quality versions of the segment. The appropriate segment quality for transmission may be selected based on prevailing network conditions. For example, if there is sufficient network bandwidth available, a high quality segment may be transmitted. As the network bandwidth decreases, it may no longer be possible to playback the video at the high quality without buffering, and as such the next segment may be transmitted at the lower quality.
It is desirable to have an additional, alternative and/or improved decoder capable of potentially improving a decoded video quality of videos having a time-varying quality.
Features, aspects and advantages of the present disclosure will become better understood with regard to the following description and accompanying drawings in which:
In accordance with the present disclosure, there is provided a method of decoding a variable quality video bitstream comprising: decoding a current frame of a current segment of the video bitstream having a first video quality; combining the decoded current frame and a decoded previous frame of an temporally previous segment of the video bitstream into an enhanced current frame, the temporally previous segment of the video bitstream having a second video quality higher than the first video quality; and decoding remaining frames of the current segment of the video bitstream using the enhanced current frame.
In an embodiment combining the decoded current frame and the decoded previous frame comprises: segmenting the decoded current frame into a plurality of non-overlapping patches; and for each patch: calculating a difference between at least a portion of the patch and a corresponding portion of the decoded previous frame; and copying the corresponding portion of the decoded previous frame to the current frame when the difference is less than a threshold.
In an embodiment combining the decoded current frame and the decoded previous frame comprises: identifying high motion areas and low motion areas between the previous frame and the current frame; copying at least a first portion of the decoded previous frame to at least a co-located portion of the low motion areas of the decoded current frame according to a first combination process; and copying at least a second portion of the decoded previous frame to at least a corresponding portion of the high motion areas of the decoded current frame according to a second combination process.
In an embodiment identifying high motion areas and low motion areas comprises: determining motion vectors between the decoded previous frame and the decoded current frame using motion estimation; segmenting the decoded current frame into a plurality of non-overlapping patches; and marking each of the plurality of patches as either a low motion patch or a high motion patch based on the motion vectors of the patch.
In an embodiment marking each of the plurality of patches comprises for each patch: averaging together the motion vectors of the respective patch to provide a patch motion vector; marking the patch as a low motion patch if the patch motion vector is less than an motion vector threshold; and marking the patch as a high motion patch if the patch motion vector is greater than or equal to the motion vector threshold.
In an embodiment the first combination process comprises: determining a difference between at least the first portion of the decoded previous frame and at least the co-located portion of the low motion areas of the current frame; copying at least the first portion of the decoded previous frame to at least the co-located portion of the low motion areas of the decoded current frame when the difference is below a threshold.
In an embodiment, the method further comprises: segmenting the low motion areas of the decoded current frame into a plurality of non-overlapping pixel patches; and for each pixel patch: determining a difference between the pixel patch and a co-located pixel patch in the decoded previous frame; and copying the co-located pixel patch from the decoded previous frame to the pixel patch of the decoded current frame when the determined difference is below a threshold.
In an embodiment the difference is determined using one of: a mean square difference; and a sum of squared differences.
In an embodiment the second combination process comprises: determining a difference between at least the second corresponding portion of the decoded previous frame and at least the corresponding portion of the high motion areas of the current frame; copying at least the first corresponding portion of the decoded previous frame to at least the portion of the low motion areas of the decoded current frame when the difference is below a threshold.
In an embodiment the second combination process further comprises: segmenting the high motion areas of the current frame into a plurality of patches; and for each patch: determining a number (Nmatch) of neighboring patches having matching motion vectors to the current patch; when Nmatch is more than a threshold, for each pixel p of the current patch: determine a corresponding pixel p′ in the decoded previous frame referenced by the motion vector of the current patch; and copying the pixel p′ to p if |p−p′|<a threshold.
In an embodiment the second combination process further comprises: segmenting the high motion areas of the current frame into a plurality of patches; and for each patch: determining a number (Nmatch) of neighboring patches having matching motion vectors to the current patch; when Nmatch is more than a threshold, determining a corresponding pixel patch P′ in the decoded previous frame referenced by the motion vector of the current patch; and copying the pixel patch P′ to the current patch P if the mean square differences (MSD) between P and P′<a threshold.
In an embodiment, the segmenting uses a patch size based on the video.
In an embodiment, the method further comprises determining the patch size by: reducing a patch size from a starting patch size and determining a variance of motion vectors of the patch size until the variance is larger than a threshold value.
In an embodiment combining the decoded current frame and the decoded previous frame comprises copying at least a portion of the decoded previous frame to the decoded current frame.
In an embodiment at least the portion of the decoded previous frame copied to the decoded current frame is processed to adjust at least one image characteristic prior to copying to the decoded current frame.
In an embodiment combining the decoded current frame and the decoded previous frame comprises combining the decoded current frame, the decode previous frame and at least one other decoded frame of the temporally previous segment of the video bitstream.
In an embodiment, the method further comprises: decoding an additional frame of the current segment of the video bitstream; and combining the decoded further frame with at least one decoded frame from the temporally previous segment to provide an enhanced additional frame.
In an embodiment the decoded previous frame combined with the decoded current frame is visually similar to the decoded current frame.
In an embodiment, the method further comprises: determining at least one frame from a plurality of frames of the temporally previous segment to use as the decoded previous frame based on a similarity to the decoded current frame.
In an embodiment, the method further comprises: decoding the immediately previous segment of the video bitstream prior to decoding the current frame of the current segment of the video bitstream.
In an embodiment the variable quality video bitstream comprises a plurality of temporal video segments, including the current segment and the temporally previous segment, each having a respective video quality.
In an embodiment each of the video segments comprises at least one intra-coded video frame that can be independently decoded and at least one inter-coded video frame that is decoded based on at least one other video frame of the video segment.
In accordance with the present disclosure, there is further provided an apparatus for decoding video comprising: a processor for executing instructions; and a memory for storing instructions, which when executed by the processor configure the apparatus to perform a method of a method of decoding a variable quality video bitstream.
In accordance with the present disclosure, there is further provided a non-transitory computer readable medium storing executable instructions for configuring an apparatus to perform a method of a method of decoding a variable quality video bitstream.
A decoder is described that uses information from a high visual quality independently encoded segment that has already been received and decoded when decoding a subsequent lower quality independently encoded segment. The decoder may improve a Quality of Experience (QoE) without incurring significant delays or additional overhead of storage and computational complexity of both the encoder and decoder, or loss of coding efficiency.
As depicted in
One possible technique to adapt to changing network conditions while streaming video content, is to split a single video into a number of consecutive segment, which may then be independently encoded at different quality level settings. The quality may then be varied for each segment, allowing the streaming quality to be adjusted based on prevailing network conditions. Each segment may vary in length, although typical segment lengths may be, for example, anywhere from between 1 second and 10 seconds. So for example, a minute long video may be encoded into 18 different encodings, such as a high quality encoding, a medium quality encoding and a low quality encoding for each of six 10 second segments. When streaming the video, the high quality version for the first 30 seconds, that is for the first three segments, may be streamed, however if the network quality degrades, the next segment may be streamed at the medium quality encoding. If the network quality continues to degrade, the last two segments may be streamed at the lowest quality encoding. Accordingly, the video will be streamed for 30 seconds at high quality, 10 seconds at medium quality and 20 seconds at low quality.
As described further below, when decoding a segment that is of a lower quality than the previous segment, the decoder may use information from the previous higher quality segment in order to improve the decoded quality of the lower quality segment.
Once the video is split into the segments 202, 204, 206, 208, each segment is encoded to provide the different quality encodings, depicted as ‘Bitrate 1’, ‘Bitrate 2’ and ‘Bitrate 3’, or which bitrate encodings 210, 212, 214 are detailed further for segment 4208. Although the following refers the to bitrate encodings 210, 212, 214 of segment 4208 it will be appreciated that the bitrate encodings for the other segments, 202, 204, 206 have a similar structure. Each of the bitrate encodings 210, 212, 214 comprises one or more group of pictures (GOP) 216, 218, 220 that encode the same frames of video at the different qualities. Each bitrate encoding is depicted as comprising 5 different GOPs. Bitrate 1 encoding 210 is of the lowest quality, bitrate 2 encoding 212 is of medium quality, and bitrate 3 encoding 214 is of the highest quality, as depicted by the relative size of the GOPs 216, 218, 220. It will be appreciated that the actual display size of a decoded video of the different bitrates may be the same.
As depicted for GOP 220, each GOP comprises a number of frames of the video 222, 224, 226, 228, 230, 232. The first frame 222 of each GOP can be decoded without reference to any other frames, and may be referred to as an intra-coded frame. The remaining frames are decoded with reference to one or more of the other frames in the GOP. For example the first frame 222 may be decoded first, followed by the second frame 224, which depends only from the first frame. The fourth frame 228, which depends only from the first frame may be decoded next, followed by the third frame 226 which depends from both the second frame 224 and the fourth frame 228. The sixth frame 232 is then decoded based on the fourth frame 228, and then the fifth frame 230 is decoded with reference to the fourth frame 228 and the sixth frame 232. As described further below, by improving the quality of a decoded reference frame used in decoding other frames, such as the first decoded frame 222, prior to decoding the remaining frames of the GOP, it is possible to improve the quality of the decoded segment. For example, the quality of the first decoded frame 222 may be improved using information from the last decoded frame of the immediately previous segment if that segment was of a higher quality than the current segment. The enhanced decoding does not require extensive modifications to the encoding process.
By extracting information contained in such a segment that is available to the decoder but was not taken advantaged by the encoder, the decoder is capable of improving the QoE of the user without incurring significant overhead to the storage and computational complexities of both the encoder and the decoder, or introducing significant delays or losses to coding efficiency.
Although the above has described the quality variations as being a result of streaming different bitrate encodings, similar variations in visual quality may also occur as a result of an encoder with a rate allocation algorithm that is not able to allocate the target bitrate in a globally optimized manner over the entire clip. This may be due to the lack of multiple pass encoding (e.g. for encoding live events) or sufficient look ahead (due to memory or delay requirements), and/or when the complexity of the input video varies significantly over time. Accordingly, when encoding segments of the video, the encoding of one segment may result in a higher or lower quality of video than the previous or subsequent segment. As such, when decoding a current segment, the previously decoded segment may be of a higher quality. The decoding of the current segment may benefit by enhancing a decoded frame of the current segment using information from the previous higher quality segment, prior to decoding the remaining frames of the segment.
When the visual quality of an input bitstream to a video decoder as described herein varies over time, at the transition from a segment with higher video quality to a temporally neighboring independently encoded segment of lower quality, last frame in display order in the higher quality segment may be referred to as a “good frame” (GE), the first intra-coded frame of the poor quality segment may be referred to as a “start frame” (SF), and the enhanced first frame used for subsequent decoding of the poor quality segment may be referred to as a “fresh start” (FS). It is noted that the SF as an intra-coded frame, was encoded without reference to the GF or any other frames in the higher quality segment.
The goal of the enhancement algorithm is to use information contained in the GF to improve the quality of the decoded SF to get an improved reference frame FS for subsequent frames in the low quality segment. Depending on the level of motion for different spatial regions of the SF, two enhancement algorithms might be used by the decoder, one for relatively low motion areas, the other for the higher motion areas. For both algorithms, the decoder will look for matches between areas in the decoded GF and the SF, as determined by a distortion metric and a threshold calculated by the decoder.
When decoding the lower quality segment 406, the first intra-coded frame 408 is decoded and the quality of the decoded frame 412 enhanced. The decoded frame 412 is enhanced by combining the frame 412 with the last frame of the high quality segment, GF 404 according to a combination process 414. The combination process 414 may copy one or more portions from the last frame of the high quality segment, GF 404, to the decoded first frame 412 to produce an enhanced first frame 416, used as a fresh start for the decoding process. The remaining frames 410 of the segment are decoded; however, with reference to the enhanced first frame 416 instead of the decoded first frame 412 as depicted by arrow 418.
Once the decoded first frame is segmented into a plurality of patches, each patch is processed (708). For each patch, a difference (Diff) between at least a portion of the patch and a corresponding portion of the decoded last frame can be calculated (710). The portion of the decoded last frame corresponding to at least the portion of the patch the difference is calculated for may be co-located or may be in a different location based on motion between the decoded last frame and the decoded first frame. With the difference calculated, it is determined if the calculated difference is below a threshold (ThDiff) (712). If the difference is not below the threshold (No at 712) the next patch (716) is processed. If the calculated difference is below the threshold (Yes at 712), the corresponding patch from the decoded last frame of the high quality segment is copied to the patch of the decoded first frame of the low quality segment (714) and the next patch processed (716). Once all of the patches have been processed, the remaining frames of the low quality segment are decoded based on the enhanced first frame (718).
For each of the high motion patches (912) the patch is segmented into sub patches (914). It is noted, that the segmenting into sub patches may not be necessary if the initial patch size is not large, such as 4×4 pixels. For each of the sub patches (916), a number of neighboring sub patches with matching motion vectors as the sub patch being processed is determined (918). It is determined if the number of neighboring sub patches with matching motion vectors (Nmatch) is greater than a threshold (920). If Nmatch is less than or equal to the threshold (No at 920) the next sub patch (926) is processed. If Nmatch is greater than the threshold (Yes at 920), it is determined which, if any, pixels from the decoded last frame should be copied to the decoded first frame (922). The determined pixels may then be copied from the decoded last frame to the corresponding portion of the decoded first frame (924) and then the next sub patch is processed (926). Once all of the sub patches are processed, the next high motion patch is processed (928). Once all of the high motion patches and the low motion patches are processed, the remaining frames of the low quality segment are decoded using the first frame enhanced with the copied portions of the last frame of the high quality segment (930).
Two specific embodiments of the decoding process described above are set out in further detail below. The first decoding embodiment is applied to HEVC encoded bitstreams and uses a patch size of 32×32 pixels for the initial segmentation. To segment the decoded first frame, SF, into high motion and low motion areas, motion estimation was conducted between the SF and the decoded last frame of the high quality segment GF at the decoder. After the motion estimate, the SF is divided into non-overlapping 32×32 pixel patches with the motion vectors (MVs) for each patch averaged and compared to a threshold ThMV. Note that each patch may overlap with multiple Prediction Units (PUs). In this embodiment ThMV was set to:
where w is the width of the video, and QP is the (average) quantization parameter of the frame. The patches whose average motion vectors are below the threshold are designated as the low motion areas, denoted as SFlow, while the rest are designated as the high motion areas, denoted by SFhi.
The low motion areas SFlow are then partitioned into non-overlapping 16×16 pixel patches. For each 16×16 patch, the Sum of Squared Differences (SSD) is calculated between the patch's pixels and the co-located pixels in the GF. If the SSD is smaller than a threshold, ThSSD, the patch in SFlow is replaced with the patch from the GF.
The performance of the decoding depends on the value of ThSSD. All integer values between 10 and 600 were exhaustively tested for ThSSD and found the threshold value ThOpt that provided the largest average peak signal to noise ratio (PSNR) gain over all frames after (and including) the SF in display order. The relationship between the values of ThOpt and the PSNR of the SF after intra encoding was plotted as depicted in
Where SAD(mv) is the Sum of Absolute Differences for my. The relationship between ThOpt and the PSNR as shown in
Th1=1.112×e(−0.2963×PSNR+15.14)−10.21, (3)
For the power function, the best fit was:
Th2=6.213×MECost1.348, (4)
From the two data fittings, the threshold ThSSD can be defined as:
ThSSD=max(Th1,Th2), (5)
Accordingly, the threshold ThSSD can be calculated given the PSNR and the MECost, which in turn can be calculated from the motion vectors calculated for the decoded first frame. The threshold ThSSD is set as the one of the two thresholds Th1 and Th2 that leads to a larger number of patches designated as “matched” in order to maximize the enhancement to the first frame provided by GF. Further, the threshold is determined based on the temporal similarity between GF and SF before encoding, represented by MECost in (4), as well as the loss of fidelity after encoding, represented by PSNR in (3).
As set out above, in order to determine the threshold ThSSD the PSNR should be known. The PSNR value for the SF after intra-frame encoding can be embedded into the HEVC bitstream, for example in SEI information or user data, by the encoder using 16 bits. Alternatively, the PSNR could be estimated at the decoder without requiring the encoder to embed the additional information.
The following is a pseudo code listing for combining the low motion areas of the first frame with corresponding areas of the decoded last frame.
The high motion areas of the decoded first frame may be enhanced from the GF. Motion information may be used in the enhancement of the high motion areas SFhi with reference to the GF. The motion vectors previously calculated by the decoder motion estimation process between the GF and the SF for the motion area segmentation and the calculations of the MECost and ThSSD may be used for the motion information when processing the high motion areas. After the motion estimation, the motion vector MV(P) for each 4×4 patch PεSFhi and its eight immediate spatially neighboring 4×4 patches. If MV(P) matched more than ThMV out of the 8 MVs from the eight 4×4 neighbors, then for each pixel pεP, the difference between p and the pixel p′ in the GF referenced by MV(P) is calculated. The difference may then be compared with a threshold ThY, with p replaced by p′ if the difference is lower than ThY. In testing, Thmv was set to 6, and values of ThY between 5 and 53 were tested using a step size of 2.
The following is a pseudo code listing for combining the low motion areas of the first frame with corresponding areas of the decoded last frame.
The decoder process described above was evaluated using an HEVC HM 8.2 encoder and the low delay configuration to encode test bitstreams. For each test clip, the HEVC encoder was ran for the first 32 frames of the clip to create the high quality segment, followed by HEVC encoding, with the same HEVC low delay configuration, of the remaining frames as the low quality segment with frame No. 33 encoded as an IDR frame SF. The QP used for encoding the first frame at the higher quality was set to be 5 levels lower than for the SF. The test clips included screen captures such as SlideEditing, video conferencing clips such as the Vidyo clips, as well as relatively higher motion clips such as the BaseketballPass and PartyScene.
The PSNR improvements for the SF, and averaged over 30 and 60 frames after (and including) the SF are given in Table 1. In the table, the values listed under the QP column are the values used for encoding the first frame of the high quality segment.
As can be seen, the PSNR improvements were significant for most of the test clips, with an average gain (with regard to all clips and bitrates) of 0.91 dB for the SF, and in most cases, a significant gain was achieved for at least 30 to 60 frames after the SF, even though the SF was the only frame to which the enhanced processing was performed. For some clips, the initial gain for the SF was lost after some frames, showing a net loss of average PSNR after 30-60 frames. This loss of the improvement to the SF over time may have occurred because after enhancing the SF, the decoder still used the same MV and residual information in the low quality bitstream for the decoding of the remaining frames in the low quality segment, even though the SF has been modified to produce the enhanced first frame used for decoding. This may lead to mismatches between the residual information needed since the enhanced SF is used as the reference, and the residual information in the bitstream, created by the encoder using the un-enhanced SF as the reference frame.
However, even with such mismatches, for many sequences, especially for video conferencing, screen capture and video surveillance applications and some clips with higher motion, a net gain was still achieved for many frames after the SF. For clips such as SlideEditing and the Vidyo clips, an average PSNR gain of well over 1 dB was observed for the entire clip after the SF, containing hundreds of frames.
As mentioned previously, the side information that can be provided from the encoder by the decoder is the PSNR for the SF after encoding as the first IDR frame of the low quality segment. This corresponds to a total of 16 bits using natural binary representation without entropy coding, and is a negligible overhead. Therefore, the PSNR gains reported reflect the “net” gains considering both the PSNR and the bitrate.
In terms of complexity, because the proposed processing was carried out for only one frame of the low quality segment, even though the decoding process involves motion estimation and calculations of SAD/SSD, the increase to the complexity of the decoding of SF is still reasonable, and lower than that for HEVC encoding of a similar frame. This is because processing required for the HEVC encoding for transform, quantization, the bulk of the processing for mode decision, and the deblocking filter are not necessary for enhanced decoding. Averaged for all frames in the low quality segment, the increase is modest considering the potential gain in PSNR and subjective quality achieved.
Finally, the clips for which a PSNR gain was not achieved in Table 1 were analyzed. In one of the clips subjective quality improvements were achieved even though the subjective quality improvements were not reflected in the PSNR. This might have been due to small mis-alignments of some pixels that might not be visible, but still have caused the PSNR to degrade. On the other hand, another clip was a case where although visible subjective improvements were achieved for both static as well as moving areas, some relatively large mis-aligned/matched patches led to an overall PSNR loss. Such mis-alignments may be visually similar to artifacts created by erroneously received motion vectors when video bitstreams are sent over error prone networks. Therefore, techniques developed for error concealment of such artifacts may be helpful in remedying such PSNR losses while preserving the gain in other areas.
In the current implementation, the value for ThY for higher motion areas was selected from the range between 5 and 53 based on the clip and bitrate. The values used for the different test clips are listed in Table 1. The value for most clips was around 5. It may be possible to determine the value for ThY by estimating the decoded PSNR.
The second decoding embodiment is applied to H.264/AVC encoded bitstreams. To segment the decoded first frame SF into high and low motion areas, motion estimation (ME) is conducted at the decoder between the SF and the decoded last frame of the high quality segment GF, with the SF divided into non-overlapping 4×4 patches with the average motion vector (MV) for each patch compared to a threshold ThMV. In this embodiment, ThMV is set to:
where w is the width of the video, and QP is the (average) quantization parameter of the frame. The patches whose average motion vectors are below the threshold are designated as the low motion areas, denoted as SFlow, while the rest are designated as the high motion areas, denoted by SFhi.
The patch size used for the initial segmentation may be determined based on the video. Two signatures of the video may be used to determine the patch size. First, ThMSD may be compared to a threshold ThMSD0=0.0377e0.2272*QP. Patches of size 32×32 were used If ThMSD<ThMSD0. Otherwise, a parameter PT was calculated at the encoder, defined as the percentage of 4×4 MVs found by the decoder between GF and SF, which led to a higher MSE than the MSE calculated with the 4×4 MVs obtained by the encoder for the same patch using the GF and the encoded input for the SF. The parameter PT calculated at the encoder may be included in the encoded bitstream or may be provided to the decoder using other channels. Then, based on the value of PT, different patch sizes were used. For example for PT between [0, 0.3%), [0.3%, 0.8%), [0.8%, 2%) and [2%, 100%), patches of 32×32, 16×16, 8×8 and 4×4 were used relatively.
The low motion areas SFlow may then be partitioned into non-overlapping patches. In this embodiment, the patch sizes used may be determined based on the frame.
For the parts where the motion is subtle and complex, the patch size should be small, while for parts where the scale of objects and motion is large, the patch size should be relatively larger. To assess the scale and complexity of motion, the variance of MVs is used to determine the patch size. First the frame is divided into 128×128 non-overlapping patches. For each patch, the variance of MVs in the patch is calculated and compared to a threshold ThV. If variance<ThV, the patch is divided into four smaller 64×64 patches and the average of MV variance in each patch is calculated. If variance<ThV, the patches are again divided. Since the average of MV variance in each patch will decrease with each division, when variance>ThV, the division of the patch size is considered proper. The following is a pseudo code listing for determining the size of the patches.
Once the frame has been segmented into patches, for each patch, the Mean Square Differences (MSD) between its pixels and their counterpart in the GF without motion compensation since it was a low motion patch is calculated. If the MSD is smaller than a threshold ThMSD the patch in SFlow is replaced with the patch in the GF.
The performance of the second embodiment depends on the value of ThMSD. The value of ThMSD was exhaustively tested with integer values between 10 and 700 and found the threshold ThOpt that provided the largest average PSNR gain over all frames after (and including) the SF in display order.
ThOPT was data fitted with AvgSAD and QP using a linear function. The best fittings were found to be:
ThMSD=−1852+54.39×QP+38.12×AvgSAD (2)
The reasoning behind using ThMSD is that the threshold ThMSD that leads to a larger number of patches designated as “matched” should be used to maximize the benefit of the presence of the GF, and the value of the thresholds should be determined by the temporal similarity between GF and SF before encoding, hence the AvgSAD in equation (2), as well as the loss of fidelity after encoding, roughly represented by QP in (2).
The following is a pseudo code listing for combining the low motion areas of the first frame with corresponding areas of the decoded last frame.
The high motion areas can be processed to enhance the SF. Motion information was used in the enhancement of the high motion areas SFhi with reference to the GF. The motion information was provided by the MVs that were obtained in the decoder ME process between the GF and the SF for the motion area segmentation and the calculations of the MECost and ThMSD. In order to improve the accuracy of the MVs after the ME, the MV(P) for each 4×4 patch PεSFhi and its eight immediate spatially neighboring 4×4 patches were compared. If MV(P) matched more than Thjudge out of the 8 neighbor MVs, then the MSD between P and the 4×4 patch P′ in the GF referenced by MV(P) was calculated. The MSD was then compared with ThMSD, and P was replaced by P′ if the difference is lower than ThMSD. Thjudge was set to 4 although other values may be used.
The following is a pseudo code listing for combining the high motion areas of the first frame with corresponding areas of the decoded last frame.
The second decoder embodiment was evaluated using the H.264x264 encoder test bitstreams. For each test clip, the x264 encoder was run for the first 10 frames of the clip to create the high quality segment, followed by x264 encoding (with the same configuration) of the remaining frames as the low quality segment with frame No. 11 encoded as an IDR frame used as the SF. The QP used for encoding the first frame of the test clip was set to be 5 levels lower than for the SF and ipratio and pbratio were set to 1. The test clips included screen captures such as SlideEditing, video conferencing clips such as the Vidyo clips, as well as relatively higher motion clips such as the Baseketball Pass and PartyScene.
The PSNR improvements for the SF, and averaged over 30 and 60 frames after (and including) the SF are given in Table 2. In the table, the values listed under the QP column are the values used for encoding the first frame of the low quality segment, that is the 11th frame of the video.
As can be seen, the PSNR improvements were significant for most of the test clips, with an average gain (with regard to all clips and bitrates) of 0.49 dB for the SF, and in most cases, a significant gain was achieved for at least 30 to 60 frames after the SF, even though the SF was the only frame to which the enhanced processing was performed. For some clips, the initial gain for the SF was lost after some frames, showing a net loss of average PSNR after 30-60 frames. This loss of the improvement to the SF over time may have occurred because after enhancing the SF, the decoder still used the same MV and residual information in the low quality bitstream for the decoding of the remaining frames in the low quality segment, even though the SF had already been modified to produce the actual reference frame of the enhanced SF. This led to mismatches between the residual information needed for the enhanced SF that was used as the reference, and the residual information in the bitstream, created by the encoder using the un-enhanced SF as the reference frame. However, even with such mismatches, for many sequences, especially for video conferencing, screen capture and video surveillance applications and some clips with higher motion, a net gain was still achieved for many frames after the SF. For clips such as SlideEditing, KristenAndSara and FourPeople, an average PSNR gain of well over 0.5 dB for the entire clip after the SF, containing hundreds of frames was observed.
The clips for which a PSNR gain was not achieved in Table 2 were analyzed. Subjective quality improvements were achieved, but were not reflected in the PSNR. This might have been due to slow-motion movements of objects with complex texture (such as leaves). Since in the disclosed decoder the slow motion patches were copied directly, the enhancement can be observed subjectively, since the motion was so small, but still results in a loss in PSNR.
Finally, in terms of complexity, because the proposed processing was carried out for only one frame of the low quality segment, even though the decoding process involves ME and calculations of SAD/MSD at the decoder, the increase to the complexity of the decoding of SF is still reasonable, and lower than that for H.264 encoding of a similar frame. This is because processing required for the H.264 encoding for transform, quantization, the bulk of the processing for mode decision, and the deblocking filter are not necessary for enhanced decoding. Averaged for all frames in the low quality segment, the increase is modest considering the potential gain in PSNR and subjective quality achieved.
Although the above has described using the decoder to improve the quality of decoded video, it may also be used to reduce the power required for encoding, as well as reducing the bandwidth required for transmitting a video. If the decoder indicates to the encoder that it is capable of the enhanced decoding described above, the encoder may vary the encoding of subsequent segments between higher and lower qualities, and the decoder may improve the decoded video quality as described above. The patch size may be fixed to reduce the computational complexity. Further, the ThMSD may be estimated using Average SAD and a different fitting such as a curve fitting. The power consumption for different test clips is shown in Table 3.
The above has described decoding video segments using various specific examples. For the sake of clarity of the description, the above has described decoding frames based on using a specific single frame, in particular the last frame of the high quality segment, for the enhancement of a single frame, in particular the first frame of the low quality segment, it is appreciated that in some cases, and especially when the video clip contains multiple scenes, the frame of the high quality segment that is used to enhance the frame of the low quality may not be temporally immediately neighboring the frame being enhanced, but rather a frame in the high quality segment that is deemed to be the most “similar” to the frame being enhanced. The similarity may be determined in various ways, such as with regard to the Sum of Absolute Differences. Accordingly, it is possible to enhance a decoded frame of a low quality segment by combining it with at least a portion of a decoded frame of a high quality segment. Further, a group of several decoded frames of the high quality segment may used to enhance one or more decoded frames of a low quality segment. Further, the above has described combining the decoded frame of the high quality segment with the decoded frame of the low quality segment by copying a portion of the decoded high quality frame to the decoded low quality frame; however, the portion of the decoded high quality frame may be processed prior to copying. Additionally or alternatively, the entire high quality frame or frames used in enhancing the decoded low quality frame or frames may be processed prior to combining. The processing may adjust one or more image characteristics of the decoded frame, such as colour, brightness, etc using different techniques such as using histogram equalization.
Although specific embodiments are described herein, it will be appreciated that modifications may be made to the embodiments without departing from the scope of the current teachings. Accordingly, the scope of the appended claims should not be limited by the specific embodiments set forth, but should be given the broadest interpretation consistent with the teachings of the description as a whole.
The system and methods described herein have been described with reference to various examples. It will be appreciated that components from the various examples may be combined together, or components of the examples removed or modified. As described the system may be implemented in one or more hardware components including a processing unit and a memory unit that are configured to provide the functionality as described herein. Furthermore, a computer readable memory, such as for example electronic memory devices, magnetic memory devices and/or optical memory devices, may store computer readable instructions for configuring one or more hardware components to provide the functionality described herein.
This application claims priority to U.S. Provisional Application Ser. No. 61/853,153 filed Mar. 30, 2013, the entire contents of which are incorporated herein by reference in their entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2014/032242 | 3/28/2014 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 61853153 | Mar 2013 | US |
