The present disclosure relates in general to video encoding and decoding.
An increasing number of applications today make use of digital video for various purposes including, for example, remote business meetings via video conferencing, high definition video entertainment, video advertisements, and sharing of user-generated videos. As technology is evolving, users have higher expectations for video quality and expect high resolution video even when transmitted over communications channels having limited bandwidth.
To permit higher quality transmission of video while limiting bandwidth consumption, a number of video compression schemes are noted including formats such as VPx, promulgated by Google Inc. of Mountain View, Calif., and H.264, a standard promulgated by ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG), including present and future versions thereof. H.264 is also known as MPEG-4 Part 10 or MPEG-4 AVC (formally, ISO/IEC 14496-10).
These compression schemes may use prediction techniques to minimize the amount of data required to transmit video information. Prediction techniques can allow for multiple past transmitted frames and future frames to be transmitted out of order and used as potential reference frame predictors for macroblocks in a frame. For example, video compression schemes, such as the MPEG or H.264 standard allow for transmission of frames out of order and use them to produce better predictors by use of forward or bidirectional prediction. Further, for example, the H.264 video compression standard allows for multiple past reference frames to be used as a predictor.
Systems, apparatuses and methods for encoding and decoding a video signal are disclosed. In accordance with one aspect of the disclosed embodiments, a method is provided for compressing a video signal having at least one frame with at least one block of pixel data. One exemplary method comprises selecting a target block in a target frame to be compressed using a processor, obtaining a first alternate reference block associated with the target block, the first alternate reference block available as a predictor for blocks of a sequence of frames and the sequence of frames defining an alternate reference period, comparing the first alternate reference block to a predictor block associated with the target block to determine whether to create a secondary alternate reference block, and, when a difference between the first alternate reference block and the predictor block is less than a threshold, selectively creating the secondary alternate reference block by selecting certain blocks from the sequence of frames that are more different from the first alternate reference block than other non-selected blocks from the sequence of frames, and creating the second alternate reference block using the selected blocks, the secondary alternate reference block available as a predictor for the blocks of the sequence of frames.
One embodiment of an apparatus for compressing a video signal having at least one frame with at least one block of pixel data taught herein comprises means for selecting a target block in a target frame to be compressed, means for obtaining a first alternate reference block associated with the target block, the first alternate reference block available as a predictor for blocks of a sequence of frames and the sequence of frames defining an alternate reference period, means for comparing the first alternate reference block to a predictor block associated with the target block to determine whether to create a secondary alternate reference block, and means for selectively creating the secondary alternate reference block when a difference between the first alternate reference block and the predictor block is less than a threshold. In this embodiment, the selectively selecting means includes means for selecting certain blocks from the sequence of frames that are more different from the first alternate reference block than other non-selected blocks from the sequence of frames and means for creating the second alternate reference block using the selected blocks, the secondary alternate reference block available as a predictor for the blocks of the sequence of frames.
Another apparatus for compressing a video signal having at least one frame with at least one block of pixel data taught herein comprises a memory and a processor configured to execute instructions stored in the memory. The instructions select a target block in a target frame to be compressed, obtain a first alternate reference block associated with the target block, the first alternate reference block available as a predictor for blocks of a sequence of frames and the sequence of frames defining an alternate reference period, compare the first alternate reference block to a predictor block associated with the target block to determine whether to create a secondary alternate reference block, and, when a difference between the first alternate reference block and the predictor block is less than a threshold, compare those blocks of the sequence of frames associated with the predictor block to select a block that is maximally different from the predictor block, open a list of blocks including the maximally different block, compare remaining blocks of the sequence of frames associated with the predictor block to the predictor block and to the maximally different block, add any block of the remaining blocks to the list of blocks when that block is more similar to the maximally different block than the predictor block, and create the second alternate reference block using the at least certain blocks of the list of blocks when a number of blocks in the list of blocks is higher than a predetermined value, the secondary alternate reference block available as a predictor for the blocks of the sequence of frames.
A method for decoding a video bitstream described herein includes selecting a current block of a current frame within the video bitstream, the video bitstream including a plurality of frames, selecting a reference frame within the video bitstream used to compress the current block, the reference frame being a frame in addition to the plurality of frames and having an alternate reference block, and decoding the current block using the alternate reference block. The alternative reference block is created by obtaining a preliminary alternate reference block associated with the current block, the preliminary alternate reference block available for inter prediction of blocks of a sequence of frames and the plurality of frames including the sequence of frames, obtaining a predictor block associated with the current block, the predictor block being an inter predicted block generated based on a frame of the plurality of frames other than the current frame, and creating the alternate reference block, responsive to a difference between the preliminary alternate reference block and the predictor block being less than a threshold, by selecting certain blocks from the sequence of frames that are more different from the preliminary alternate reference block than other non-selected blocks from the sequence of frames, and creating the alternate reference block using the selected blocks.
An apparatus for decoding a video bitstream described herein includes a memory and a processor. The processor is configured to execute instructions stored in the memory to select a current block of a current frame within the video bitstream, the video bitstream including a plurality of frames and select a reference frame within the video bitstream used to compress the current block, the reference frame being a frame in addition to the plurality of frames and having an alternate reference block created by obtaining a preliminary alternate reference block associated with the current block, the preliminary alternate reference block available for inter prediction of blocks of a sequence of frames and the plurality of frames including the sequence of frames, obtaining a predictor block associated with the current block, the predictor block being an inter predicted block generated based on a frame of the plurality of frames other than the current frame, and creating the alternate reference block, responsive to a difference between the preliminary alternate reference block and the predictor block being less than a threshold. The processor is configured to create the alternate reference block by selecting certain blocks from the sequence of frames that are more different from the preliminary alternate reference block than other non-selected blocks from the sequence of frames, and creating the alternate reference block using the selected blocks. Finally, the processor is configured to decode the current block using the alternate reference block.
Variations of these methods and apparatuses and details of other aspects and embodiments are described in additional detail hereinafter.
The description herein makes reference to the accompanying drawings wherein like reference numerals refer to like parts throughout the several views, and wherein:
Encoder 20 includes a temporal filter 40, which as shown in
When input video stream 10 is presented for encoding, each frame 16 within input video stream 10 is processed in units of macroblocks. At intra/inter prediction stage 26, each macroblock can be encoded using either intra-frame prediction (i.e., within a single frame) or inter-frame prediction (i.e. from frame to frame). In either case, a prediction macroblock can be formed. In the case of intra-prediction, a prediction macroblock can be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction macroblock can be formed from samples in one or more previously constructed reference frames as described in additional detail herein.
Next, still referring to
The reconstruction path in
Other variations of encoder 20 can be used to encode compressed bitstream 24. For example, a non-transform based encoder can quantize the residual signal directly without transform stage 28. In another embodiment, an encoder may have quantization stage 30 and dequantization stage 34 combined into a single stage.
The encoding process shown in
When compressed bitstream 24 is presented for decoding, the data elements within compressed bitstream 24 can be decoded by entropy decoding stage 46 (using, for example, Context Adaptive Binary Arithmetic Decoding) to produce a set of quantized transform coefficients. Dequantization stage 48 dequantizes the quantized transform coefficients, and inverse transform stage 50 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the reconstruction stage in the encoder 20. Using header information decoded from the compressed bitstream 24, decoder 42 can use intra/inter prediction stage 52 to create the same prediction macroblock as was created in encoder 20. At the reconstruction stage 54, the prediction macroblock can be added to the derivative residual to create a reconstructed macroblock. The loop filter 56 can be applied to the reconstructed macroblock to reduce blocking artifacts. Deblocking filter 58 can be applied to the reconstructed macroblock to reduce blocking distortion, and the result is output as output video stream 44.
Other variations of decoder 42 can be used to decode compressed bitstream 24. For example, a decoder may produce output video stream 44 without deblocking filtering stage 58.
Referring again to encoder 20, video encoding methods compress video signals by using lossless or lossy compression algorithms to compress each frame or blocks of each frame of a series of frames. As can be implied from the description above, intra-frame coding refers to encoding a frame using data from that frame, while inter-frame coding refers to predictive encoding schemes such as schemes that comprise encoding a frame based on other so-called “reference” frames. For example, video signals often exhibit temporal redundancy in which frames near each other in the temporal sequence of frames have at least portions that match or at least partially match each other. Encoders can take advantage of this temporal redundancy to reduce the size of encoded data by encoding a frame in terms of the difference between the current frame and one or more reference frames.
Video encoders may use motion compensation based algorithms that match blocks of the frame being encoded to portions of one or more other frames. The block of the encoded frame may be shifted in the frame relative to the matching portion of the reference frame. This shift is characterized by a motion vector. Any differences between the block and partially matching portion of the reference frame may be characterized in terms of a residual. The encoder 20 may thus encode a frame as data that comprises one or more of the motion vectors and residuals for a particular partitioning of the frame. A particular partition of blocks for encoding the frame may be selected by approximately minimizing a cost function that, for example, balances encoding size with distortion to the content of the frame resulting from encoding.
As described briefly above, many video coding algorithms first partition each picture into macroblocks. Then, each macroblock can be coded using some form of predictive coding method. Some video coding standards use different types of predicted macroblocks in their coding. In one scenario, a macroblock may be one of three types: 1) Intra (I) macroblock that uses no information from other pictures in its coding; 2) Unidirectionally Predicted (P) macroblock that uses information from one preceding picture; and 3) Bidirectionally Predicted (B) macroblock that uses information from one preceding picture and one future picture.
To facilitate higher quality compressed video, it is helpful to have the best matching reference frame in order to have the smallest difference to encode, which generally results in a more compact encoding. Reference frames are based on past frames, future frames, or an intra-frame so that the encoder can find the best matching block to use in the predictive process as shown in, for example, U.S. Application Publication No. 2005/0286629 A1. More recently, reference frames based on synthesized or constructed frames that are not shown to the end user after decoding have been introduced. Such reference frames are shown in, for example, co-pending U.S. Application Publication No. 2010/0061461 A1.
Another method of constructing a reference frame can include selecting a target frame and using temporal filter 40 (
Regardless of the method used to create these reference frames, such constructed reference frames are referred to herein as “alternate reference frames.” The alternate reference frames are constructed from macroblocks, which are referred to herein as “alternate reference blocks.” In addition to these alternate reference frames, two other types of reference frames are used in prediction, particularly the example described below. The first is the last frame before the current frame to be encoded. The second is a golden frame, which is most simply defined as one frame's worth of decompressed data from an arbitrarily distant past. As frames 14 in video sequence 12 progress, a new golden frame can be designated, often but not necessarily periodically. Alternatively, parts of the golden frame are updated as frames 14 in video sequence 12 progress.
A block and/or frame can be encoded using any one of these reference frames so as to minimize the cost of encoding a frame or block where the cost is, for example, a comparison of bit count to reconstruction accuracy performed by techniques known in the art. An alternate reference frame can, however, end up being exactly or almost exactly the same as the last or golden frame. That is, the creation of the alternate reference frame sometimes creates areas of prediction that are no better for any frame than the last or golden frame's predictor for the areas. Such an alternate reference frame is not an ideal predictor—that is, it is not useful for those areas. Embodiments described herein describe a mechanism for creating a useful “secondary” reference frame in such situations. Namely, blocks of an alternate reference frame that provide prediction no better for a frame than the last or golden frame are replaced with or supplemented by predictors that provide a better alternative for a specified number of frames in a sequence of frames. Herein, these are called secondary alternate reference blocks.
Such a mechanism is illustrated first with reference first to
For each frame 60a-60i, macroblocks 62a-62i are identified, which are located at the same corresponding spatial position in each frame. Spatial correspondence, as referred herein, means a pixel, macroblock or other partition has the same or corresponding position from one frame to another, and spatial corresponding blocks are used in the example below. However, the blocks used for the comparisons and calculations are not limited to blocks in the respective frames that spatially correspond to each other. The spatial positions of the selected blocks may be different because of motion compensation or other predictive techniques (such as using a constructed reference frame of a different size from the frames to be encoded). Accordingly, the comparisons occur between blocks that are associated with each other, whether that includes spatial correspondence or correspondence with motion compensation and/or consideration of the predictive technique. Motion compensation, for example, can be used selectively where the target macroblock was encoded in a preceding pass with an intra mode, an inter mode with a non zero motion vector or with 0,0 inter-frame coding where the correlation is below a predetermined threshold.
An alternate reference period 64 is also shown in
Alternate reference frame 66 includes an alternate reference block 68 spatially corresponding to macroblocks 62a-62f. Golden frame 74 includes a golden frame reference block 76 spatially corresponding to macroblocks 62a-62f.
A secondary alternate reference frame 70 is also shown next to alternate reference frame 66. Similar to alternate reference frame 66, secondary alternate reference frame 70 includes a secondary alternate reference block 72 spatially corresponding to macroblocks 62a-62f. Alternately, a secondary alternate reference frame 70 may not be completely created, but only one or more secondary alternate reference blocks such as secondary alternate reference block 72 may be created. The creation of secondary alternate reference blocks is detailed further hereinafter. For application to inter-prediction encoding and decoding, if a secondary alternate reference block is created, it will replace the spatially corresponding alternate reference block in the alternate reference frame 66. But alternatively, secondary alternate reference blocks may be stored and used as predictors separately from the predictor blocks of alternate reference frame 66.
However, if a secondary alternate reference block 72 would be advantageous (80), then the “most different” block(s) in alternate reference period 64 are selected (82). The term “most different” may encompass a number of different algorithms capable of selecting one or more blocks within alternate reference period 64 that are less similar to alternate reference block 68 than those that are not selected. For example, a fixed number of the least similar blocks may be selected. Alternatively, the half of the blocks in the alternate reference period 64 that are most dissimilar could be selected.
Once the “most different” block(s) are selected (82), then any selected blocks are combined to create secondary alternate reference block 72 (84). This combination could be accomplished by blurring. For example, blurring may be performed by averaging each spatially corresponding individual pixel from each of the selected blocks. The blur may also utilize a weighted average where certain selected blocks may have a higher weighting than others. Other ways of combining the blocks, if desirable, can be used. After secondary alternate reference block 72 is created (84), the method ends. Secondary alternate reference block 72 replaces original alternate reference block 68, or alternatively, secondary alternate reference block 72 is added as an additional possible predictor block for blocks to be encoded. As explained in detail below, this process can be repeated for each block of alternate reference frame 66 and/or for each block of a frame to be encoded.
In the process shown in
As shown in
Once the alt ref block is created/selected (102), the golden frame reference block associated with the target block is selected (104). In this example, this would be golden frame reference block 76, as it is the block in golden frame 74 that spatially corresponds with macroblock 62b.
After the golden frame reference block is selected (104), processing initializes a variable maxgfdiff to zero (106). The variable maxgfdiff is also called error maxgfdiff and its use is described in more detail hereinafter.
The difference between the alt ref block and the golden frame reference block is then calculated and compared to a threshold (108). The difference may be calculated using the sum of absolute differences (SAD) algorithm based on a pixel-by-pixel comparison. The difference may be alternatively be calculated using other methods, including but not limited to the sum of absolute transformed differences (SATD) or the sum of squared differences (SSD). The threshold is an arbitrary number that may be a constant or set by the encoder 20 using various criteria, including but not limited to block size and the difference algorithm used. Once calculated, if the difference is greater than the threshold, then the process ends and no alternative to the alt ref block is created for use by encoder 20 for spatially corresponding target blocks in an alternate reference period, in this example at least blocks 62b-62f in alternate reference period 64.
If instead the difference between the alt ref block and the golden frame reference block is less than the threshold (108), processing selects a first available frame within the frames of interest (112), generally those in the alternate reference period, to continue the process of determining a secondary alternate reference block. A macroblock that spatially corresponds with the target macroblock is selected from the selected frame for the next processing steps.
Once the macroblock is selected from the first available frame (112), a difference, error gfdiff, between the golden frame reference block and the selected macroblock is calculated (114). As disclosed with respect to the calculation of differences above (108), the difference may be calculated here using SAD or other methods. Error gfdiff is then compared to error maxgfdiff (116).
If error gfdiff is greater than error maxgfdiff (116), then error maxgfdiff is set as the current value of error gfdiff (118). The currently selected macroblock is then selected as the maximally different block (120). If instead error gfdiff is less than error maxgfdiff (116), error maxgfdiff remains unchanged. Whether error maxgfdiff is updated or not, processing advances to determine whether there are additional frames available in the alternate reference period (122). If there are additional frames available (122), the next frame in the alternate reference period and its macroblock spatially corresponding to the target macroblock are selected (112) to repeat the steps of calculating error gfdiff (114) and comparing it to error maxgfdiff (116) to determine whether to update error maxgfdiff and the maximally different block. Once no additional frames are available (122), processing initializes a counter variable, countwins, to zero (124). At the end of this processing, the maximally different block of all spatially corresponding blocks in the sequence of frames is known.
Now with reference to
That is, the first available frame within the sequence of frames is again selected, as is the macroblock in that frame that spatially corresponds to the target macroblock (130).
Once a macroblock is selected (130), error gfdiff variable is again calculated for the selected macroblock (132). As mentioned above, error gfdiff variable is calculated as the difference between the golden frame reference block and the currently-selected macroblock using SAD or other methods. Next, a variable afdiff is calculated as the difference between the maximally different block and the currently-selected macroblock (134). As described previously, the difference may be calculated using SAD or other methods.
Variable afdiff is next compared with error gfdiff (136). If the value of variable afdiff is less than the value of error gfdiff, counter countwins variable is incremented (138). The current selected macroblock is then added to a list of macroblocks that can be used to create the secondary alternate reference block (140), where the list preferably already includes the maximally different block. If instead the value of variable afdiff is greater than or equal to the value of error gfdiff (136), processing advances without incrementing counter countwins and adding to the list.
Regardless of whether variable afdiff is less than error gfdiff or not (136), a check is made as to whether there are any additional frames available to process in the alternate reference period (142). Until there are no additional frames (142), processing continues by selecting the next frame and associated macroblock (130), calculating error gfdiff and variable afdiff (132, 134) and comparing them (136) to determine whether or not to increment counter countwins (138) and to add the current macroblock to the list (140).
If no additional frames are available (142), the value of counter countwins is compared to a threshold value (144). The threshold value is an arbitrary number that is used to determine whether creating the secondary alt ref block should be done or not. That is, if the number of blocks is not sufficiently high, as represented by counter countwins, the additional bits needed to encode yet another alt ref block in the bitstream are not considered to be worth the benefit of having that alt ref block. The threshold value may be a constant, such as 2 blocks, or be set by the encoder 20 using various criteria, including but not limited to encoder quality settings and previous prediction results.
If the value of counter countwins is less than or equal to the threshold value (144), the process ends. No secondary alt ref block is created, and the unaltered alt ref block remains available as one option for encoder 20 to encode blocks. In contrast, if the value of counter countwins is greater than the threshold value (144), the selected blocks in the list are combined, such as by blurring or in other ways as described previously, to create the secondary alt ref block (146). For example, the maximally different block could be used as the center block of a plurality of selected blocks in a process using a temporal filter. The process then ends.
As mentioned briefly above, the secondary alt ref block so created can replace the spatially corresponding alt ref block originally selected (102). Alternatively, the secondary alt ref block could be added as one more possible predictor block for blocks to be encoded. Note that once the process is completed for a target macroblock, it does not have to be repeated for any spatially corresponding block in the sequence of frames considered. The sec alt reference block was created in consideration of these blocks and is available to them for encoding.
The process above can be repeated for each block or only certain blocks of the original alternate reference frame for the alternate reference period if the target macroblock does not encompass the entirety of the target frame.
The preceding technique can include optional features. For example, although all of the blocks in the list are used to create the secondary alt ref block (146), the worst two or three could be used, such as by averaging. Alternatively, the worst block (i.e., the maximally different block) could be used alone or even omitted from the list. Further, the alt ref block could be replaced only after trying the new secondary alt ref block and determining that it provides better prediction for blocks over several frames, such as two or three frames.
The process described above is implemented in two-pass encoder 20 as described. However, it could be implemented in a single-pass encoder where a number of input frames are buffered in the encoder before that encoder produces a first compressed frame. The information collected during the first pass of a two-pass encode process (e.g., encoding modes of macroblocks) can be calculated over the set of buffered frames rather than the entire clip as it is in the two-pass encoder.
The operation of encoding can be performed in many different ways and can produce a variety of encoded data formats. The above-described embodiments of encoding or decoding may illustrate some exemplary encoding techniques. However, in general, encoding and decoding are understood to include any transformation or any other change of data whatsoever.
Encoder 20 and/or decoder 42 are implemented in whole or in part by one or more processors which can include computers, servers, or any other computing device or system capable of manipulating or processing information now-existing or hereafter developed including optical processors, quantum processors and/or molecular processors. Suitable processors also include, for example, general purpose processors, special purpose processors, IP cores, ASICS, programmable logic arrays, programmable logic controllers, microcode, firmware, microcontrollers, microprocessors, digital signal processors, memory, or any combination of the foregoing. In the claims, the term “processor” should be understood as including any the foregoing, either singly or in combination. The terms “signal” and “data” are used interchangeably.
Encoder 20 and/or decoder 42 also include a memory, which can be connected to the processor through, for example, a memory bus. The memory may be read only memory or random access memory (RAM) although any other type of storage device can be used. Generally, the processor receives program instructions and data from the memory, which can be used by the processor for performing the instructions. The memory can be in the same unit as the processor or located in a separate unit that is coupled to the processor.
For example, encoder 20 can be implemented using a general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms and/or instructions described herein.
Encoder 20 and decoder 42 can, for example, be implemented in a wide variety of configurations, including for example on servers in a video conference system. Alternatively, encoder 20 can be implemented on a server and decoder 42 can be implemented on a device separate from the server, such as a hand-held communications device such as a cell phone. In this instance, encoder 20 can compress content and transmit the compressed content to the communications device, using the Internet for example, as shown in
The operations of encoder 20 or decoder 42 (and the algorithms, methods, instructions etc. stored thereon and/or executed thereby) can be realized in hardware, software or any combination thereof. All or a portion of embodiments of the present invention can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example tangibly contain, store, communicate, and/or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or a semiconductor device. Other suitable mediums are also available.
The above-described embodiments have been described in order to allow easy understanding of the present invention and do not limit the present invention. On the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structure as is permitted under the law.
This application is a continuation of co-pending U.S. patent application Ser. No. 13/081,668, filed Apr. 7, 2011.
Number | Name | Date | Kind |
---|---|---|---|
4816906 | Kummerfeldt et al. | Mar 1989 | A |
4924310 | von Brandt | May 1990 | A |
5148269 | de Haan et al. | Sep 1992 | A |
5337086 | Fujinami | Aug 1994 | A |
5371841 | Jones | Dec 1994 | A |
5389068 | Keck | Feb 1995 | A |
5398068 | Liu et al. | Mar 1995 | A |
5442458 | Rabbani et al. | Aug 1995 | A |
5461423 | Tsukagoshi | Oct 1995 | A |
5483287 | Siracusa | Jan 1996 | A |
5485279 | Yonemitsu et al. | Jan 1996 | A |
5512952 | Iwamura | Apr 1996 | A |
5568200 | Pearlstein et al. | Oct 1996 | A |
5576767 | Lee et al. | Nov 1996 | A |
5586285 | Hasbun et al. | Dec 1996 | A |
5686962 | Chung et al. | Nov 1997 | A |
5717394 | Schwartz et al. | Feb 1998 | A |
5731840 | Kikuchi et al. | Mar 1998 | A |
5748789 | Lee et al. | May 1998 | A |
5767909 | Jung | Jun 1998 | A |
5818536 | Morris et al. | Oct 1998 | A |
5886742 | Hibi et al. | Mar 1999 | A |
5912676 | Malladi et al. | Jun 1999 | A |
5926226 | Proctor et al. | Jul 1999 | A |
5991447 | Eifrig et al. | Nov 1999 | A |
5999641 | Miller et al. | Dec 1999 | A |
6005980 | Eifrig et al. | Dec 1999 | A |
6075875 | Gu | Jun 2000 | A |
6084912 | Reitmeier et al. | Jul 2000 | A |
6108383 | Miller et al. | Aug 2000 | A |
6115076 | Linzer | Sep 2000 | A |
6181822 | Miller et al. | Jan 2001 | B1 |
6222174 | Tullis et al. | Apr 2001 | B1 |
6236682 | Ota et al. | May 2001 | B1 |
6272179 | Kadono | Aug 2001 | B1 |
6289049 | Kim et al. | Sep 2001 | B1 |
6292837 | Miller et al. | Sep 2001 | B1 |
6327304 | Miller et al. | Dec 2001 | B1 |
6359929 | Boon | Mar 2002 | B1 |
6370267 | Miller et al. | Apr 2002 | B1 |
6381277 | Chun et al. | Apr 2002 | B1 |
6473463 | Agarwal | Oct 2002 | B2 |
6560366 | Wilkins | May 2003 | B1 |
6658618 | Gu et al. | Dec 2003 | B1 |
6661842 | Abousleman | Dec 2003 | B1 |
6711211 | Lainema | Mar 2004 | B1 |
6735249 | Karczewicz et al. | May 2004 | B1 |
6774924 | Kato et al. | Aug 2004 | B2 |
6774929 | Kopp | Aug 2004 | B1 |
6909749 | Yang et al. | Jun 2005 | B2 |
6985527 | Gunter et al. | Jan 2006 | B2 |
7027654 | Ameres et al. | Apr 2006 | B1 |
7050503 | Prakash et al. | May 2006 | B2 |
7085319 | Prakash et al. | Aug 2006 | B2 |
7221710 | Lee | May 2007 | B2 |
7253831 | Gu | Aug 2007 | B2 |
7406053 | Cheung et al. | Jul 2008 | B2 |
7430261 | Forest et al. | Sep 2008 | B2 |
7499492 | Ameres et al. | Mar 2009 | B1 |
7529199 | Wijnands et al. | May 2009 | B1 |
7532808 | Lainema | May 2009 | B2 |
7606310 | Ameres et al. | Oct 2009 | B1 |
7671894 | Yea et al. | Mar 2010 | B2 |
7728840 | Hung | Jun 2010 | B2 |
7734821 | Wang et al. | Jun 2010 | B2 |
7773677 | Lee | Aug 2010 | B2 |
7974233 | Banerjee | Jul 2011 | B2 |
8005137 | Han et al. | Aug 2011 | B2 |
8111752 | Kumar et al. | Feb 2012 | B2 |
8284846 | Lamy-Bergot et al. | Oct 2012 | B2 |
8310521 | Zhang et al. | Nov 2012 | B2 |
8638854 | Bankoski et al. | Jan 2014 | B1 |
9014266 | Gu et al. | Apr 2015 | B1 |
20010022815 | Agarwal | Sep 2001 | A1 |
20020031272 | Bagni et al. | Mar 2002 | A1 |
20020036705 | Lee et al. | Mar 2002 | A1 |
20020071485 | Caglar et al. | Jun 2002 | A1 |
20020118295 | Karczewicz et al. | Aug 2002 | A1 |
20020172289 | Akiyoshi et al. | Nov 2002 | A1 |
20030081850 | Karczewicz et al. | May 2003 | A1 |
20030123545 | Prakash et al. | Jul 2003 | A1 |
20030165331 | Van Der Schaar | Sep 2003 | A1 |
20030202594 | Lainema | Oct 2003 | A1 |
20030215014 | Koto et al. | Nov 2003 | A1 |
20040013308 | Jeon et al. | Jan 2004 | A1 |
20040037357 | Bagni et al. | Feb 2004 | A1 |
20040042549 | Huang et al. | Mar 2004 | A1 |
20040080669 | Nagai et al. | Apr 2004 | A1 |
20040184533 | Wang | Sep 2004 | A1 |
20040202252 | Lee | Oct 2004 | A1 |
20040228410 | Ameres et al. | Nov 2004 | A1 |
20050008240 | Banerji et al. | Jan 2005 | A1 |
20050031030 | Kadono et al. | Feb 2005 | A1 |
20050123056 | Wang et al. | Jun 2005 | A1 |
20050147167 | Dumitras et al. | Jul 2005 | A1 |
20050185045 | Kamariotis | Aug 2005 | A1 |
20050207490 | Wang et al. | Sep 2005 | A1 |
20050226321 | Chen | Oct 2005 | A1 |
20050259736 | Payson | Nov 2005 | A1 |
20050286629 | Dumitras et al. | Dec 2005 | A1 |
20060050149 | Lang et al. | Mar 2006 | A1 |
20060050695 | Wang | Mar 2006 | A1 |
20060062481 | Suvanto | Mar 2006 | A1 |
20060083300 | Han et al. | Apr 2006 | A1 |
20060126734 | Wiegand et al. | Jun 2006 | A1 |
20060159174 | Chono | Jul 2006 | A1 |
20060198443 | Liang et al. | Sep 2006 | A1 |
20060285598 | Tulkki | Dec 2006 | A1 |
20070009034 | Tulkki | Jan 2007 | A1 |
20070019730 | Lee et al. | Jan 2007 | A1 |
20070092010 | Huang et al. | Apr 2007 | A1 |
20070109409 | Yea et al. | May 2007 | A1 |
20070130755 | Duquette et al. | Jun 2007 | A1 |
20070177665 | Zhou et al. | Aug 2007 | A1 |
20070199011 | Zhang et al. | Aug 2007 | A1 |
20070201559 | He | Aug 2007 | A1 |
20070206673 | Cipolli et al. | Sep 2007 | A1 |
20070211798 | Boyce et al. | Sep 2007 | A1 |
20070230563 | Tian et al. | Oct 2007 | A1 |
20080112486 | Takahashi et al. | May 2008 | A1 |
20080115185 | Qiu et al. | May 2008 | A1 |
20080130755 | Loukas et al. | Jun 2008 | A1 |
20080219351 | Kim et al. | Sep 2008 | A1 |
20080273599 | Park et al. | Nov 2008 | A1 |
20080317138 | Jia | Dec 2008 | A1 |
20090028247 | Suh et al. | Jan 2009 | A1 |
20090103610 | Puri | Apr 2009 | A1 |
20090122859 | Yasuda | May 2009 | A1 |
20090147856 | Song et al. | Jun 2009 | A1 |
20090148058 | Dane et al. | Jun 2009 | A1 |
20090154563 | Hong et al. | Jun 2009 | A1 |
20090175330 | Chen et al. | Jul 2009 | A1 |
20090238269 | Pandit et al. | Sep 2009 | A1 |
20090238277 | Meehan | Sep 2009 | A1 |
20100061444 | Wilkins et al. | Mar 2010 | A1 |
20100061461 | Bankoski et al. | Mar 2010 | A1 |
20100061645 | Wilkins et al. | Mar 2010 | A1 |
20100086027 | Panchal et al. | Apr 2010 | A1 |
20100104016 | Aoki | Apr 2010 | A1 |
20100195721 | Wu et al. | Aug 2010 | A1 |
20100239015 | Wang et al. | Sep 2010 | A1 |
20110069751 | Budagavi | Mar 2011 | A1 |
20110090960 | Leontaris et al. | Apr 2011 | A1 |
20110164684 | Sato et al. | Jul 2011 | A1 |
20120063513 | Grange et al. | Mar 2012 | A1 |
20120092452 | Tourapis et al. | Apr 2012 | A1 |
20120189058 | Chen et al. | Jul 2012 | A1 |
20120257677 | Bankoski et al. | Oct 2012 | A1 |
20120328005 | Yu et al. | Dec 2012 | A1 |
20130022099 | Liu et al. | Jan 2013 | A1 |
20130044817 | Bankoski et al. | Feb 2013 | A1 |
20130114695 | Joshi et al. | May 2013 | A1 |
20130242046 | Zhang et al. | Sep 2013 | A1 |
20130279589 | Gu et al. | Oct 2013 | A1 |
20140169449 | Samuelsson et al. | Jun 2014 | A1 |
Number | Date | Country |
---|---|---|
0634873 | Jan 1995 | EP |
1351510 | Oct 2003 | EP |
1496706 | Jan 2005 | EP |
2403618 | Jan 2005 | GB |
8280032 | Oct 1996 | JP |
11262018 | Sep 1999 | JP |
11289544 | Oct 1999 | JP |
11313332 | Nov 1999 | JP |
11513205 | Nov 1999 | JP |
100213018 | Aug 1999 | KR |
20010030916 | Apr 2001 | KR |
WO0150770 | Jul 2001 | WO |
WO03026315 | Mar 2003 | WO |
WO03084235 | Oct 2003 | WO |
WO2006078115 | Jul 2006 | WO |
WO2008008331 | Jan 2008 | WO |
WO2011005624 | Jan 2011 | WO |
Entry |
---|
Shen, L., et al. Fast mode decision for multiview video coding, 2009, IEEE, entire document. |
Yang, H. et al., Optimizing Motion Compensated Prediction for Error Resilient Video Coding, IEE Transactions on Image Processing, vol. 19, No. 1 Jan. 2010, entire document. |
Athanasios, et al.,“Weighted prediction methods for improved motion compensation,” Image Processing (ICIP), 2009 16th IEEE International Conference, Nov. 7, 2009, pp. 1029-1032. |
Chen, Michael C., et al.; “Design and Optimization of a Differentially Coded Variable Block Size Motion Compensation System”, IEEE 1996, 4 pp. |
Chen, Xing C., et al.; “Quadtree Based Adaptive Lossy Coding of Motion Vectors”, IEEE 1996, 4 pp. |
Chong Soon Lim et al. Reference Lists for B Pictures Under Low Delay Constraints, 95. MPEG Meeting; Jan. 24, 2011; Jan. 21, 2011. |
EP127356814 Search Report mailed Oct. 30, 2014. |
Zhang, Kui, et al.; “Variable Block Size Video Coding With Motion Prediction and Motion Segmentation”, SPIE vol. 2419, 1995, 9 pp. |
Guillotel, Philippe, et al.; “Comparison of motion vector coding techniques”, SPIE vol. 2308, 1994, 11 pp. |
IPRP for PCT/US2013/037058 from related application mailed Nov. 6, 2014. |
ISR & Written Opinion, Re: Application # PCT/US2012/043748;Sep. 14, 2012. |
Karczewicz, Marta, et al.; “Video Coding Using Motion Compensation With Polynomial Motion Vector Fields”, IEEE COMSOC EURASIP, First International Workshop on Wireless Image/Video Communications—Sep. 1996, 6 pp. |
Kim, Jong Won, et al.; “On the Hierarchical Variable Block Size Motion Estimation Technique for Motion Sequence Coding”, SPIE Visual Communication and Image Processing 1993, Cambridge, MA, Nov. 8, 1993, 29 pp. |
Liu, Bede, et al.; “A simple method to segment motion field for video coding”, SPIE vol. 1818, Visual Communications and Image Processing 1992, 10 pp. |
Liu, Bede, et al.; “New Fast Algorithms for the Estimation of Block Motion Vectors”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, No. 2, Apr. 1993, 10 pp. |
Luttrell, Max, et al.; “Simulation Results for Modified Error Resilient Syntax With Data Partitioning and RVLC”, ITU—Telecommunications Standardization Sector, Study Group 16, Video Coding Experts Group (Question 15), Sixth Meeting: Seoul, South Korea, Nov. 2, 1998, 34 pp. |
Martin, Graham R., et al.; “Reduced Entropy Motion Compensation Using Variable Sized Blocks”, SPIE vol. 3024, 1997, 10 pp. |
Nicolas, H., et al.; “Region-based motion estimation using deterministic relaxation schemes for image sequence coding”, IEEE 1992, 4 pp. |
Nokia, Inc., Nokia Research Center, “MVC Decoder Description”, Telecommunication Standardization Sector, Study Period 1997-2000, Geneva, Feb. 7, 2000, 99 pp. |
ON2 Technologies Inc., White Paper TrueMotion VP7 Video Codec, Jan. 10, 2005, 13 pages, Document Version: 1.0, Clifton Park, New York. |
Orchard, Michael T.; “Exploiting Scene Structure in Video Coding”, IEEE 1991, 5 pp. |
Orchard, Michael T.; “Predictive Motion-Field Segmentation for Image Sequence Coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, No. 1, Feb. 1993, 17 pp. |
Schiller, H., et al.; “Efficient Coding of Side Information in a Low Bitrate Hybrid Image Coder”, Signal Processing 19 (1990) Elsevier Science Publishers B.V. 61-73, 13 pp. |
Schuster, Guido M., et al.; “A Video Compression Scheme With Optimal Bit Allocation Among Segmentation, Motion, and Residual Error”, IEEE Transactions on Image Processing, vol. 6, No. 11, Nov. 1997, 16 pp. |
Steliaros, Michael K., et al.; “Locally-accurate motion estimation for object-based video coding”, SPIE vol. 3309, 1997, 11 pp. |
Stiller, Christoph; “Motion-Estimation for Coding of Moving Video at 8 kbit/s with Gibbs Modeled Vectorfield Smoothing”, SPIE vol. 1360 Visual Communications and Image Processing 1990, 9 pp. |
Strobach, Peter; “Tree-Structured Scene Adaptive Coder”, IEEE Transactions on Communications, vol. 38, No. 4, Apr. 1990, 10 pp. |
Wang et al., On reference picture list construction for uni-predicted partitions, JCT-VC Meeting, MPEG Meeting, Geneva, Mar. 11, 2011. |
Wiegand, Thomas, et al.; Long-Term Memory Motion-Compensated Prediction, date unknown. |
Wiegand, Thomas, et al.; “Rate-Distortion Optimized Mode Selection for Very Low Bit Rate Video Coding and the Emerging H.263 Standard”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 6, No. 2, Apr. 1996, 9 pp. |
Wright, R. Glenn, et al.; “Multimedia—Electronic Technical Manual for ATE”, IEEE 1996, 3 pp. |
Bo Hong: “Introduction to H.264”, Internet citation, XP002952898, pp. 5, 14-15. |
Borman S. et al., “Super-Resolution From Image Sequences—A Review”, Proceedings of Midwest Symposium on Circuits and Systems, pp. 374-378, Aug. 9, 1998. |
Carreira, Joao et al. “Constrained Parametric Min-Cuts for Automatic Object Segmentation”, 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, Jun. 13-18, 2010. |
Feng Wu et al, “Efficient Background Video Coding with Static Sprite Generation and Arbitrary-Shape Spatial Prediction Techniques”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, No. 5, pp. 394-405, May 1, 2003. |
Girod B. et al., “3-D Image Models and Compression: Synthetic Hybrid or Natural Fit?”, International Conference on Image Processing, vol. 2, pp. 525-529, Oct. 24, 1999. |
Ebrahimi, Touradj, et al.; “Joint motion estimation and segmentation for very low bitrate video coding”, SPIE vol. 2501, 1995, 12 pp. |
Hiroshi Watanabe et al, “Sprite Coding in Object-Based Video Coding Standard: MPEG-4”, Proceedings of Multiconference on Systemics, Cybernetics and Informatics, vol. 13, pp. 420-425, Jul. 1, 2001. |
International Search Report and Written Opinion for International Patent Application PCT/US2012/031579; dated Jun. 21, 2012. |
International Search Report dated Apr. 20, 2010 from the corresponding International Application No. PCT/US2009/056448. |
International Search Report for related matter PCT/US2013/037058 mailed Dec. 16, 2013. |
Irani M et al, “Video Compression Using Mosaic Representations”, Signal Processing Image Communication, vol. 7 No. 4., pp. 529-552, Nov. 1, 1995. |
“Liu, P., et al, ““A fast and novel intra and inter modes decision prediction algorithm for H.264/AVC based-on the characteristics of u macro-block””, 2009 Fifth Inti. Confr. on Intelligent Information Hiding and Multimedia Signal Processing, pp. 286-289,http:/ /i eeexp lore. ieee .o rg/s tamp/stamp .jsp ?tp=&arn umber =533 7 483”. |
ON2 Technologies, Inc., White Paper On2's TrueMotion VP7 Video Codec,Jan. 10, 2005, pp. 13 pages, Document Version:1.0, Clifton Park, New York. |
Park, Jun Sung, et al., “Selective Intra Prediction Mode Decision for H.264/AVC Encoders”, World Academy of Science, Engineering and Technology 13, (2006). |
Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video, Video coding for low bit rate communication, International Telecommunication Union, ITU-T Recommendation H.263, Feb. 1998, 167 pp. |
Written Opinion of the International Searching Authority dated Aug. 8, 2010, from the corresponding International Application No. PCT/US2009/056448 filed Sep. 10, 2009. |
Zhi Liu, Zhaoyang Zhang, Liquan Shen, Mosaic Generation in H.264 Compressed Domain, IEEE 2006. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video; Advanced video coding for generic audiovisual services”. H.264. Version 1. International Telecommunication Union. Dated May 2003. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video; Advanced video coding for generic audiovisual services”. H.264. Version 3. International Telecommunication Union. Dated Mar. 2005. |
“Overview; VP7 Data Format and Decoder”. Version 1.5. On2 Technologies, Inc. Dated Mar. 28, 2005. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video; Advanced video coding for generic audiovisual services”. H.264. Amendment 1: Support of additional colour spaces and removal of the High 4:4:4 Profile. International Telecommunication Union. Dated Jun. 2006. |
“VP6 Bitstream & Decoder Specification”. Version 1.02. On2 Technologies, Inc. Dated Aug. 17, 2006. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video”. H.264. Amendment 2: New profiles for professional applications. International Telecommunication Union. Dated Apr. 2007. |
“VP6 Bitstream & Decoder Specification”. Version 1.03. On2 Technologies, Inc. Dated Oct. 29, 2007. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video”. H.264. Advanced video coding for generic audiovisual services. Version 8. International Telecommunication Union. Dated Nov. 1, 2007. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video”. H.264. Advanced video coding for generic audiovisual services. International Telecommunication Union. Version 11. Dated Mar. 2009. |
“Series H: Audiovisual and Multimedia Systems; Infrastructure of audiovisual services—Coding of moving video”. H.264. Advanced video coding for generic audiovisual services. International Telecommunication Union. Version 12. Dated Mar. 2010. |
“Implementors' Guide; Series H: Audiovisual and Multimedia Systems; Coding of moving video: Implementors Guide for H.264: Advanced video coding for generic audiovisual services”. H.264. International Telecommunication Union. Version 12. Dated Jul. 30, 2010. |
“VP8 Data Format and Decoding Guide”. WebM Project. Google On2. Dated: Dec. 1, 2010. |
Bankoski et al. “VP8 Data Format and Decoding Guide; draft-bankoski-vp8-bitstream-02” Network Working Group. Dated May 18, 2011. |
Bankoski et al. “Technical Overview of VP8, An Open Source Video Codec for the Web”. Dated Jul. 11, 2011. |
Bankoski, J., Koleszar, J., Quillio, L., Salonen, J., Wilkins, P., and Y. Xu, “VP8 Data Format and Decoding Guide”, RFC 6386, Nov. 2011. |
Mozilla, “Introduction to Video Coding Part 1: Transform Coding”, Video Compression Overview, Mar. 2012, 171 pp. |
Number | Date | Country | |
---|---|---|---|
Parent | 13081668 | Apr 2011 | US |
Child | 14158958 | US |