Patent Grant
8831099
Embodiments according to the present invention generally relate to video encoding (video compression).
H.264, also known as Moving Pictures Experts Group-4 (MPEG-4) Part 10 or MPEG-4 Advanced Video Coding (AVC), is a standard for video compression (encoding). A video includes a series of pictures (or frames), with each frame consisting of a two-dimensional array of pixels. The pixels are divided into macroblocks (e.g., a 16×16 array of pixels). A macroblock number or identifier (ID) is associated with each macroblock. In general, the macroblocks are numbered starting at the top-left of the frame, in increasing order from left-to-right and top-to-bottom.
H.264 offers a rich choice of macroblock encoding modes or types. The encoding modes can be generally characterized as either inter-encoding, which takes advantage of temporal redundancy between frames, or intra-encoding, which takes advantage of spatial redundancy between macroblocks in the same frame. A decision regarding the type of encoding to be used is made by comparing the rate-distortion (RD) cost for each of the different modes under consideration, and then selecting the mode that has the lowest cost.
Macroblocks are typically encoded in a multi-stage pipeline. A macroblock (e.g., macroblock K) is encoded in one stage and then reconstructed from the encoded data in a subsequent stage. For intra-encoding, the mode decision process for macroblock K utilizes the reconstructed data for one or more of the macroblocks (e.g., macroblock K-1) that preceded macroblock K in the encoding pipeline.
In some cases, a macroblock needed for the mode decision process may still be in the process of being reconstructed, and hence the necessary reconstructed data is not available for the cost calculations. Consequently, the encoding pipeline may stall until the data needed for the cost calculations becomes available.
According to embodiments of the present invention, a decision about how to encode a particular macroblock (the “current macroblock”) is made using raw (non-encoded) data for the current macroblock in the cost calculations instead of reconstructed data from another macroblock. For example, instead of using the last (right-most) column of reconstructed pixel data for the preceding macroblock (e.g., the macroblock to the left of the current macroblock), the first (left-most) column of raw pixel data for the current macroblock is used in the mode decision process. Similarly, the bottom row of reconstructed pixel data for the macroblock above the current macroblock can be replaced in the mode decision process with the top row of raw pixel data for the current macroblock.
Because it is not necessary to wait for reconstructed data to become available for the mode decision process, related delays in the encoding pipeline are eliminated and encoder performance is improved. Also, the need to store and retrieve reconstructed data for the cost calculations is eliminated. As such, less memory is consumed and the amount of bus traffic is reduced.
These and other objects and advantages of the various embodiments of the present invention will be recognized by those of ordinary skill in the art after reading the following detailed description of the embodiments that are illustrated in the various drawing figures.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.
Reference will now be made in detail to embodiments in accordance with the present invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with these embodiments, it will be understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims.
Furthermore, in the following detailed description of embodiments of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be recognized by one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the embodiments of the present invention.
Some portions of the detailed descriptions, which follow, are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer-executed step, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as “accessing,” “computing,” “selecting,” “using,” “comparing,” “encoding,” “generating” or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Furthermore, the term “adjacent” is used herein to describe either a pair of vertices that have common edges or a pair of edges that have a common vertex, and the term “contiguous” is used herein to describe adjacent pixels connected together so as to form a row or column of pixels.
Embodiments of the invention, such as those described by the flowcharts of
By way of example, and not limitation, computer-usable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory or other memory technology, compact disk ROM (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information.
Communication media can embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
In one embodiment, the encoding pipeline is utilized in an H.264 encoder that, in general, encodes (compresses) frames of image data (e.g., video data). H.264 may also be known as MPEG-4 Part 10 or MPEG-4 AVC. According to H.264, each video or picture frame is divided into macroblocks (an M×N array of pixels).
In the present embodiment, some type of macroblock ID is associated with each macroblock. Each macroblock ID is unique within a frame; that is, each macroblock in a frame can be uniquely identified by its macroblock ID. The macroblocks in a frame are arrayed in rows and columns. In one embodiment, encoding is performed starting with the macroblock in the first row and first column—that is, with reference to the orientation in which the image would be displayed, encoding starts at the left end of the top row of the image and proceeds in order from left-to-right and top-to-bottom. The macroblock IDs usually coincide with the encoding order.
Raw (non-encoded) image data (e.g., video data) is encoded using some type of compression scheme (e.g., H.264) to generate a set of encoding parameters such as, but not limited to, frame type (e.g., intra-coded I-frame, predicted P-frame or bi-predicted B-frame), macroblock prediction mode (e.g., inter-encoding versus intra-encoding), transform (e.g., discrete cosine transform) coefficients, texture coefficients, and motion vector information. The encoding parameters are then translated into variable length codes (VLCs), using Huffman coding for example. The encoded bitstream is a serialized bitstream containing the VLCs.
One of the functions performed by the encoding pipeline is to select a macroblock prediction mode (inter-encoding versus intra-encoding) for each macroblock of the raw data. A decision regarding the type of encoding to be used is made by comparing the rate-distortion (RD) cost for each of the modes and then selecting the mode that has the lowest cost. RD costs can be determined using any of a variety of cost functions known in the art, such as sum of absolute differences (SAD) or sum of absolute transformed differences (SATD) cost functions.
In the example of
As part of the mode decision process, the first stage of the
In a conventional pipeline, reconstructed data from MB K-1 in the second stage would be used in the mode decision process for MB K in the first stage. In contrast, according to embodiments of the invention, the raw data accessed by the first stage is used in the mode decision process instead of the reconstructed data. In other words, according to embodiments of the invention, it is not necessary to wait for reconstructed data for a preceding macroblock (e.g., macroblock K-1, which is at the second pipeline stage) in order to make an encoding mode decision for the current macroblock K (which is at the first pipeline stage). As such, RD costs can be calculated at the first pipeline stage instead of at the second pipeline stage. As a result, the encoder can perform more efficiently.
In one embodiment, the macroblock 200 includes a 16×16 array of pixels or blocks, although the present invention is not so limited. For example, instead of a 16×16 array, the macroblock 200 may include a 4×4 array of blocks. The macroblock 200 does not necessarily have to be square; for example, the macroblock 200 may include a 16×8 array of blocks. In general, the macroblock 200 includes an M×N array of macroblocks.
In
At the point during processing that is represented in
According to conventional methods, the reconstructed pixels used in the mode decision process include some combination of the following pixels, depending on the macroblock encoding mode being evaluated:
the pixels related to the last (bottom) row of the macroblock 220—pixels p(15,0), p(15,1), . . . , p(15,15);
the pixels related to the last (right-most) column of the macroblock 230—pixels q(0,15), q(1,15), . . . , q(15,15); and/or
the last pixel of the last (bottom) row of the macroblock 210—pixel r(15,15).
In contrast to conventional methods, embodiments according to the invention use raw pixel data for the current macroblock (macroblock 200), instead of the reconstructed pixel data listed above, to make a decision about how to encode the current macroblock. More specifically, instead of using the data for the pixels in the last (bottom) row of the macroblock 220 in certain cost calculations, data for the pixels in the first (top) row of the macroblock 200 is used. Similarly, instead of using data for the pixels in the last (right-most) column of the macroblock 230 in certain other cost calculations, data for the pixels in the first (left-most) column of the macroblock 200 is used. Also, instead of using data for the last pixel of the last (bottom) row of the macroblock 210 in yet other cost calculations, data for the first pixel of the first (top) row of the macroblock 200 is used. It is understood that some cost calculations may use combinations of the aforementioned pixel data.
In one embodiment, there are at least four different luma intra-encoding prediction modes considered during the mode decision process—these are referred to herein as horizontal prediction mode, vertical prediction mode, DC (direct current) prediction mode, and plane prediction mode. The RD cost associated with each of these modes is determined using different combinations of the raw data for the current macroblock.
The particular raw data used in the cost calculation for each of these modes is summarized in Table 1 below and discussed further in conjunction with
DC={(c(0,0)+c(0,1)+ . . . +c(0,15))+(c(0,0)+c(1,0)+ . . . +c(15,0))}>>5. (1)
For the RD cost calculation, the DC value calculated using equation (1) is applied across the entire macroblock as shown in
Although each of the examples above is based on a 16×16 array of pixels per macroblock, the present invention is not so limited. For example, the mode decision process described herein can be readily extended to 4×4 luma prediction modes.
The RD cost can be calculated for each of the other H.264 intra-encoding prediction modes using the current macroblock's raw pixel data in a manner similar to that just described. In other words, generally speaking, the cost calculations are preformed as they usually are, except that the current macroblock's raw data is substituted for the corresponding reconstructed data—that is, a row of raw data is substituted for a row of reconstructed data, and so on. The RD cost associated with inter-encoding the current macroblock is calculated in a conventional manner.
In block 620 of
In block 630, in one embodiment, the cost (Pcost) for inter-encoding the current macroblock is determined using the reconstructed data from the second image frame mentioned above.
In block 640, the costs calculated in blocks 620 and 630 are compared to determine which is the lowest. In one embodiment, the minimum value of Icost-H, Icost-V and Icost-DC is compared to the value of Pcost. If Pcost is lower than the minimum of Icost-H, Icost-V and Icost-DC, then the current macroblock is inter-encoded. Otherwise, the current macroblock is intra-encoded.
Note that the comparison in block 640 is intended to decide between inter-encoding and intra-encoding and is not necessarily performed to select a type of intra-encoding (if intra-encoding is to be performed), although the comparison could be used in this manner. In other words, if Icost-H, for example, was determined to have the lowest value of all the computed costs (including Pcost), this would not necessarily mean that the current macroblock will be encoded using the horizontal prediction mode, although it could be.
In block 720 of
Significantly, as previously described herein, the intra-encoding costs are computed using non-encoded (raw) data from the first (current) macroblock in place of reconstructed image data from at least a second macroblock of the image frame (e.g., one or more of the macroblocks 210, 220 and 230 of
In general, as one example, an intra-encoding cost can be computed using, at least in part, non-encoded image data from a first set of contiguous pixels of the first macroblock instead of reconstructed image data from a second set of contiguous pixels of another macroblock, where the first and second sets of pixels are adjacent and parallel. More specifically, the first set of contiguous pixels may include a row (e.g., the top row) of the first macroblock and the second set of contiguous pixels may include a row (e.g., the bottom row) of another macroblock, or the first set of contiguous pixels may include a column (e.g., the left column) of the first macroblock and the second set of contiguous pixels may include a column (e.g., the right column) of another macroblock.
As another example, an intra-encoding cost can be computed using, at least in part, non-encoded image data for a first pixel (e.g., pixel c(0,0) of
As yet another example, an intra-encoding cost can be computed using a DC value that is itself computed using non-encoded image data from a column of the first macroblock in combination with non-encoded image data from a row of the first macroblock (using equation (1), for example).
In block 730 of
Although specific steps are disclosed in the flowcharts 600 and 700, such steps are exemplary. That is, embodiments of the present invention are well-suited to performing various other steps or variations of the steps recited in the flowcharts.
In summary, according to embodiments of the invention, a decision about how to encode a particular macroblock is made using raw data for that macroblock in the cost calculations instead of reconstructed data from one or more other macroblocks. Thus, it is not necessary to wait for reconstructed data to become available for the mode decision process. Consequently, encoder performance is improved because stalls due to the unavailability of reconstructed data are avoided. Also, because the need to store and retrieve reconstructed data for the cost calculations is eliminated, less memory and less bus bandwidth are consumed.
The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5859668 | Aono et al. | Jan 1999 | A |
| 5930390 | Coelho | Jul 1999 | A |
| 6025888 | Pauls | Feb 2000 | A |
| 6765964 | Conklin | Jul 2004 | B1 |
| 7170937 | Zhou | Jan 2007 | B2 |
| 7289674 | Karczewicz | Oct 2007 | B2 |
| 7706442 | Kuo | Apr 2010 | B2 |
| 7953284 | Au et al. | May 2011 | B2 |
| 8059717 | Saigo et al. | Nov 2011 | B2 |
| 8315467 | Nakamura et al. | Nov 2012 | B2 |
| 8355449 | Moccagatta et al. | Jan 2013 | B2 |
| 8437405 | Moccagatta et al. | May 2013 | B1 |
| 8761253 | Garg et al. | Jun 2014 | B2 |
| 20020159525 | Jeong | Oct 2002 | A1 |
| 20030158608 | Ishikawa et al. | Aug 2003 | A1 |
| 20030223645 | Sun et al. | Dec 2003 | A1 |
| 20030235249 | Zhao et al. | Dec 2003 | A1 |
| 20040028282 | Kato et al. | Feb 2004 | A1 |
| 20040114576 | Itoh et al. | Jun 2004 | A1 |
| 20040184548 | Kerbiriou et al. | Sep 2004 | A1 |
| 20050089094 | Yoo et al. | Apr 2005 | A1 |
| 20050157797 | Gaedke | Jul 2005 | A1 |
| 20050249293 | Zeng et al. | Nov 2005 | A1 |
| 20060018385 | Lee | Jan 2006 | A1 |
| 20060126730 | Arakawa et al. | Jun 2006 | A1 |
| 20060233447 | Matsubara | Oct 2006 | A1 |
| 20060251330 | Toth et al. | Nov 2006 | A1 |
| 20060268990 | Lin et al. | Nov 2006 | A1 |
| 20070019726 | Cha et al. | Jan 2007 | A1 |
| 20070217508 | Shimada et al. | Sep 2007 | A1 |
| 20070229325 | Yokose et al. | Oct 2007 | A1 |
| 20080069211 | Kim et al. | Mar 2008 | A1 |
| 20080219576 | Jung et al. | Sep 2008 | A1 |
| 20080232463 | Lu et al. | Sep 2008 | A1 |
| 20080291995 | Graham et al. | Nov 2008 | A1 |
| 20090034856 | Moriya et al. | Feb 2009 | A1 |
| 20090154560 | Hong et al. | Jun 2009 | A1 |
| 20100158105 | Garg et al. | Jun 2010 | A1 |
| 20100166073 | Schmit et al. | Jul 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 2007013298 | Jan 2007 | JP |
| 2008227670 | Sep 2008 | JP |
| 1020060027795 | Mar 2006 | KR |
| 2004104930 | Dec 2004 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20100150237 A1 | Jun 2010 | US |
