Patent Application
20120257675
The present disclosure relates generally to video processing and more particularly to video encoding.
Video encoding allows compressed video signals to be transmitted with reduced bandwidth and to be stored in smaller portions of memory. The degree of compression and other processing can affect the quality and resolution of a video image decoded from and encoded video signal. Accordingly, the video signal being encoded is typically encoded with sufficient information to provide for the display resolution, frame rate, and other quality related parameters that can be reproduced by a display device having specific processing and display capabilities. However, in some situations it is desirable to provide a common video stream to different devices that support various processing and display characteristics to limit the amount of processing that needs to be done at the display. For example, in video conferencing, it can be useful to provide each conference node with a copy of a common video signal so that all conference participants are viewing the same video. However, the display devices at the various nodes can have different processing and display capabilities. One approach is to encode the video signal to have the quality parameters associated with the most capable display device and let each of the less capable devices reprocess the video signal to accommodate a lesser video quality. However, this can result in inefficient use of communication bandwidth, as well as place an undesirable processing load on each receiving device.
The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.
To illustrate, an SVC video stream is a video information stream that includes a base layer and one or more enhancement layers. The base layer can be decoded individually to determine a base video stream having minimum quality characteristics, such as resolution and frame rate, associated with the SVC video stream. The enhancement layers can be decoded, in combination with the base layer, to determine a video stream that has enhanced quality characteristics relative to the base video stream. As explained further herein, the base layer and the enhancement layers can be encoded using different respective encoding paths. As used herein, an encoding path refers to the set of processes employed to encode video information, and can include hardware encoding modules, software encoding modules, and any combination thereof. As used herein, a hardware encoding module is a set of hardware that is dedicated to performing one or more processes, such as intra-prediction, motion search, motion refinement, mode decision, motion vector differential, transform and quantization, and the like, employed in encoding a video stream. As used herein, a hardware encoding module does not refer to a general purpose processor programmed to perform a video encoding process, nor does it refer to modules that perform general data communication, data management, memory transfers, or other overhead functions of the video encoder.
Video encoder 100 includes a multiplexer 102, an upscaler 103, an encoder control module 104, a base/enhancement control module 105, a hardware encoder 106, a software entropy encoder 107, a base/enhancement control module 108, a hardware entropy encoder 109, and a base/enhancement layer combiner 110. The multiplexer 102 includes a plurality of inputs, whereby each input receives a corresponding video stream, labeled “S1”, “S2”, “S3”, and “S4.” Multiplexer 102 also includes an output connected to base/enhancement control module 105 and a control input connected to encoder control module 104. Base/enhancement control module 105 includes connections to upscaler 103 and hardware encoder 106. Upscaler 103 includes a connection to an input of the multiplexer 102 to provide the video stream S4. Hardware encoder 106 includes a connection to base/enhancement control module 108, which includes connections to software entropy encoder 107 and hardware entropy encoder 109. Base/enhancement layer combiner 110 also includes connections to software entropy encoder 107 and hardware entropy encoder 109.
The video streams S1-S3 are video information streams extracted by the video encoder 100 from a received video signal (not shown). In an embodiment, the video signal can represent video information stored in a recording medium, such as an optical or magnetic disk, magnetic tape, or other media. In another embodiment, the video signal can be received via a communication network, such as a local area network or wide area network. Each of the streams S1-S3 are composed of a series of image frames, and each image frame is composed of a set of pixels. Multiplexer 102 is configured to provide a frame of a selected one of the video streams S1-S4 at its output based on information received at the control input.
The encoder control module 104 provides control information to the control input of the multiplexer 102 to select the next frame for encoding. In particular, by selecting a designated stream via the control information, encoder control module 104 causes multiplexer 104 to provide a frame of the designated video stream at its output. In an embodiment, encoder control module 104 selects the video streams in a round robin fashion. In another embodiment, the encoder control module can determine which streams have frames available for processing, and select those streams in a pre-defined or reprogrammable order.
The base/enhancement control module 105 is configured to route a received frame based on whether the frame is to be encoded in the base layer of the SVC video stream or an enhancement layer of the SVC video stream. The base/enhancement control module 105 can make the determination based on information received from a control module (not shown), such as a processor, based on information in the received frame itself, based on the time the frame was received, based on the location of the frame in the associated video stream, or a combination thereof. In response to determining the received frame is to be encoded in the base layer, base/enhancement control module 105 provides the frame to the hardware encoder 106. In response to determining the received frame is to be encoded in an enhancement layer, the base/enhancement control module 105 determines whether the frame has been upscaled, based on information received from a control module, information stored within the frame itself, or a combination thereof. If it determines that the frame has not been upscaled, the base/enhancement control module 105 provides the frame to the upscaler 103. If the base/enhancement control module 105 determines that the frame has been upscaled, it provides the frame to the hardware encoder 106.
The upscaler 103 is configured to upscale a received frame, thereby increasing the number of pixels that represent the frame. Upscaler 103 can upscale by interpolating pixel values, by repeating pixel values, or by using another upscaling technique. The upscaling technique and amount of upscaling can be predefined, can be indicated by control information received from a control module, or can be indicated by information contained within the frame.
Hardware encoder 106 includes one or more hardware encoding modules to encode received frames into encoded video information. In an embodiment, the hardware encoder encodes the received frames in accordance with a specified encoding technique, such as the H.264 encoding standard. In another embodiment, the encoding technique employed by the hardware encoder 106 is reprogrammable, and is based on control information provided by a control module.
The base/enhancement control module 108 is configured to route received encoded video information based on whether the received information is enhancement layer information or base layer information. In response to determining the received encoded video information is enhancement layer information, the base/enhancement control module 108 provides the encoded video information to the software entropy encoder 107. In response to determining the received encoded video information is base layer information, the base/enhancement control module 108 provides the encoded video information to the hardware entropy encoder 109.
Software entropy encoder 107 is a software routine executed on a processor (not shown). The software entropy encoder 107 entropy encodes received encoded video information according to an associated entropy encoding technique. Hardware entropy encoder 109 is a hardware module dedicated to entropy encoding received encoded video information according to an associated entropy encoding technique. In an embodiment, the entropy encoding implemented by the software entropy encoder 107 differs from the entropy encoding by the hardware entropy encoder 109. For example, the software entropy encoder 107 and hardware entropy encoder 109 can implement different entropy encoding processes, or can implement the same entropy encoding process with different encoding parameters.
The base/enhancement layer combiner combines received enhancement layer information and base layer information to form an SVC video stream. The SVC video stream is provided at an output of the video encoder 110 for provision to a communication interface or for storage.
In operation, the video encoder 102 encodes base layer information and enhancement layer information using two different encoding paths. In particular, the video encoding path for base layer information includes the hardware encoder 106 and the hardware entropy encoder 109. In contrast, the video encoding path includes the hardware encoder 106 and the software entropy encoder 107, and does not include the hardware entropy encoder 109. Accordingly, both of the video encoding paths include the hardware encoder 106. By employing the hardware encoder 106 for encoding of both the base layer and the enhancement layers of an SVC video stream, the efficiency of the encoding process is improved. For example, conventional SVC encoders typically employ software, rather than dedicated software modules to encode enhancement layer information. This can slow the encoding process relative to using dedicated hardware modules.
The control module 220 receives image frames and routes video information based on the frames to one or both of the MS module 221 and the IP module 222. The MS module 221 is configured to process image frames received from the control module 220 based on a segmentation into macroblocks of pixel values, such as of 16 pixels by 16 pixels size, from the columns and rows of a frame and/or other control information. In an embodiment, the MS module 221 determines, for each macroblock or macroblock pair of a frame one or more motion vectors that represents the displacement of the macroblock (or subblock) from a reference frame or reference field of the video signal to a current frame or field. In operation, the motion search module operates within a search range to locate a macroblock (or subblock) in the current frame or field to an integer pixel level accuracy such as to a resolution of 1-pixel. Candidate locations are evaluated based on a cost formulation to determine the location and corresponding motion vector that have a most favorable (such as lowest) cost.
The motion refinement module 223 is configured to generate a refined motion vector for each macroblock of the plurality of macroblocks, based on a received motion search motion vector. In an embodiment the MR module determines, for each macroblock or macroblock pair of a frame, a refined motion vector that represents the displacement of the macroblock from a reference frame or reference field of the video signal to a current frame or field.
Based on the pixels and interpolated pixels, the MR module 223 refines the location of the macroblock in the current frame or field to a greater pixel level accuracy such as to a resolution of ¼-pixel or other sub-pixel resolution. Candidate locations are also evaluated based on a cost formulation to determine the location and refined motion vector that have a most favorable (such as lowest) cost
The IP module 222 generates an intra-prediction mode for each macroblock of the plurality of macroblocks. In an embodiment IP module 222 operates as defined by the H.264 standard, however, other intra-prediction techniques can likewise be employed. In particular, intra-prediction module 222 evaluates a plurality of IP modes such as an Intra-4.times.4 or Intra-16×16, which are luma prediction modes, chroma prediction (8×8) or other intra coding, based on motion vectors determined from neighboring macroblocks to determine the intra-prediction mode and the associated cost.
The mode decision module 212 determines a final macroblock cost for each macroblock of the plurality of macroblocks based on costs associated with the refined motion vector, the direct mode motion vector, and the best intra prediction mode, and in particular, the method that yields the lowest cost, or an otherwise specified cost.
The MVD 226 module is configured to determine differences between received motion vectors. The differential information can be employed for subsequent entropy encoding of the encoded video information.
The TQZ module 220 generates an encoded video signal by transforming, coding, quantizing, and zig-zag encoding residual pixel values into quantized transformed coefficients that can be further coded, such as by entropy coding. The decoder 230 is configured to decode the encoded video information provided by the TQZ module 227 and provide the decoded video information to the MS module 221.
The decoder 230 is a video decoder module configured to decode video information provided by the TQZ module 227 and provide the decoded video information to the MS module 221. This allows the MS module 221 to reduce errors in the video encoding. In the illustrated embodiment, the decoder 230 includes an upscaler 231, which is employed by the decoder 230 to perform video decoding. In an embodiment, the upscaler 231 employs the same hardware modules as the upscaler 103 of
In an embodiment, the hardware encoding modules illustrated in
Other embodiments, uses, and advantages of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The specification and drawings should be considered exemplary only, and the scope of the disclosure is accordingly intended to be limited only by the following claims and equivalents thereof.
