1. Technical Field
The present invention relates to a statistical multiplexer for coding and multiplexing multiple channels of digital television data.
2. Related Art
Digital television has become increasingly popular due to the high quality video image it provides, along with informational and entertainment features, such as pay-per-view, electronic program guides, Internet hyperlinks, and so forth. Such television data can be communicated to a user, for example, via a broadband communication network, such as a satellite or cable television network, or via a computer network. The video data can include high definition (HD) and standard-definition (SD) television (TV).
However, due to the bandwidth limitations of the communication channel, it is necessary to adjust a bit rate of the digital video programs that are encoded and multiplexed for transmission in a single compressed bit stream. A goal of such bit rate adjustment is to meet the constraint on the total bit rate of the multiplexed stream, while also maintaining a satisfactory video quality for each program.
Accordingly, various statistical multiplexers have been developed that evaluate statistical information of the source video that is being encoded, and allocate bits for coding the different video channels. For example, video channels that have hard-to-compress video, such as a fast motion scene, can be allocated more bits, while channels with relatively easy to compress scenes, such as scenes with little motion, can be allocated fewer bits.
In MPEG-2 and MPEG-4 digital video systems, the complexity of a video frame is measured by the product of the quantization level (QL) used to encode that frame and the number of bits used for coding the frame (R). This means the complexity of a frame is not known until it has been encoded. As a result, the complexity information always lags behind the actual encoding process, which requires the buffering of a number of frames prior to encoding, thereby adding expense and complexity. This kind of look-behind information may be avoided by using some pre-encoding statistics about the video, such as motion estimation (ME) scores or a need parameter (NP) to provide a complexity measure. However, the relationship between the pre-encoding statistics of a video frame and the complexity of that frame may not be direct, and sometimes the relationship may change due to the changing subject matter of the source video.
Previous statistical multiplexing systems employed a number of individual encoders that encode data from a number of incoming channels of source video data. The system dynamically allocated bits to the individual encoders to encode frames of video data from the channels. The system used pre-encoding statistics of the source video frames that are closely related to the complexity of the frames, and account for changing content in the source video, to dynamically allocate bits. With more channels included in video content and increasing density of the data in high density systems it is desirable to continue to improve the performance of such multiplexing systems.
Embodiments of the present invention provide improvements to a statistical multiplexer (statmux) system for coding and multiplexing multiple channels of digital television data.
The statmux system according to embodiments of the present invention includes a plurality of encoders, a statmux controller and a multiplexer. The encoders each receive a separate video channel input and provide complexity information for individual video channels to be compressed in the form of a need parameter (NP) to the statmux controller. The statmux controller, or rate control processor, collects the NP data from all the video channels and assigns an encoding bandwidth to each video channel in the form of an encoding bit rate. Each channel encoder receives a different encoding bit rate based on its NP in relation to the needs of all the other channels. The encoders use the encoding bit rate to perform video compression and deliver the compressed frames at the assigned bit rate to the multiplexer which combines the channels into a single video signal.
A particular method for allocating bits used by the statmux controller for encoding a plurality of channels of video data sources includes: obtaining a NP for a current picture for each channel and providing an encoding bit rate to each encoder for allocating the data for each channel according to the NP.
Embodiments of the present invention particularly consider when a scene change occurs in determining the NP. After a scene change occurs, the NP data provided by the encoder may no longer be an accurate determination for complexity of the video data to be encoded. In the past, when determining a complexity the encoder looked only at the current picture and previous picture history. However, if a new scene is significantly more complex and requires a higher bit rate, the data complexity determination based only on current or previous data may not be adequate.
Accordingly, embodiments of the present invention consider complexity information from the new scene occurring after the scene change as well as data from the current scene in providing the NP data for the statmux controller. To ensure enough data is included from the new scene, the complexity determination in one embodiment evaluates all of the remaining data of the Look Ahead Buffer (LAB) after the scene change occurs.
To determine complexity in the new scene, an evaluation of parameters such as the I, P and B type pictures in the video data, is made. The I, P and B type pictures can code very differently, so it is important to group complexity calculations by picture type. The percentage of pictures grouped according to each of the I, P and B types are, thus, used to determine the overall complexity. In some embodiments, only the new and current scenes are evaluated to determine complexity. Data from the “previous scene,” where the previous scene is the scene immediately preceding the current scene, is not included in the complexity determination. In some embodiments, the code further considers a particular class of “difficult frames.” Difficult frames can occur in circumstances such as when a picture flashes, as during an explosion, and these frames are bit intensive and complex to encode.
Further details of the present invention are explained with the help of the attached drawings in which:
The encoded data provided to a multiplexer (mux) 8 is combined into a single bitstream that is provided to a transport packet buffer 10. The transport packet buffer 10 then provides the compressed and multiplexed video channels to a transmitter 12 for transmission to a remote receiver that will decode and provide the individual channels to a television or other video display device.
The encoders 41-4N can be either for standard definition (SD) television or high definition (HD) television. A block diagram of an SD encoder 20 is shown in
A block diagram of the HD encoder 30 is shown in
Both the SD encoder of
Control information such as the NP and ST are exchanged between the encoders and the statmux controller to control the Bitrate Queue (BRQ) in each controller for the system to maximize efficiency. For the NP, each encoder will provide the NP information to the statmux controller 6 to indicate the difficulty of the content being encoded. The statmux controller 6 will then use this NP data to determine what ratio of bits to give each encoder. For ST, each encoder will receive state information from the statmux controller 6. This ST is updated with the BRQ data in regular intervals, for example every 2 milliseconds. The ST information can include a minimum bitrate, nominal bitrate and a command that can be set to hold the bitrate constant. There is a separate BRQ for each encoder that will contain equally spaced bitrates to be applied. As mentioned above, an example of the BRQ application period is 2 miliseconds. In one example to enable efficient operation all PCI bus accesses by the statmux controller and encoders are via writes. No PCI reads are performed, so data is always stored at the destination. Further information about statistical data determination, such as NP and BRQ, are described in more detail to follow.
Both the SD and HD encoders can be combined into a single statmux system, such as that illustrated in
The system of
A key part of a statistically multiplexed multi-channel encoding system of the invention is the calculation of NP. The visual characteristics and complexity information regarding the source video are collected and condensed into this single parameter, which is referred to as the “need parameter.” The NP is calculated for each video channel, and is updated once per frame whenever a new video frame is processed by an encoder. Optionally, the NP can be updated more often, such as multiple times per frame. Moreover, for field-picture mode, the NP can be updated once per field.
The current frame motion estimation (ME) scores, average frame ME scores and current frame activity are preferably directly applied in the calculation of the NP. Optionally, a table look-up may be used. The NP calculation functions in an encoder provide the NP according to the current picture type in the beginning of a new frame (such as HD or SD), and pass the NP to the statmux controller. The NP must arrive at the statmux controller no later than, e.g., two quantization level/bit rate (QL/BR) cycles before the start of the slice encoding at the encoder. This lead time ensures the statmux controller has enough processing time for bandwidth allocation.
During operation of a statmux system, such as illustrated in
For the high-definition encoder that processes multiple panels of a frame in parallel, such as illustrated in
As the statmux controller 6 receives updated NP data, it reallocates the bandwidths for all the video services based on the latest information. The bandwidth allocation is sent back to each encoder, such as 41-4N of
The statmux controller keeps an approximate Video Buffering Verifier (VBV) model for each encoder, such as is known from the MPEG standard, to ensure that each frame from each encoder is encoded within acceptable size limits. The VBV model is only approximate because the actual transmission rate changes that occur at the decode time of a frame cannot be precisely modeled in advance, at the time of encoding. The statmux controller 6 also keeps a bit accurate model of the BRQ, and calculates the minimum and maximum limits on the transmission rate before any transmission rate change is issued. Since all the video services need not be frame-synchronized, the encoding bit rates and transmission rates are updated as frequently as the statmux controller 6 can handle.
Initially, the encoder 41 includes a video capture module 50. The video capture module 50 provides a Video Input Clock signal, illustrated in
The encoder 41 further includes a Pre-Look Ahead Buffer (Pre-LAB) 52 that receives the output of the video capture module 50. The PreLAB 52 includes a few pipeline states before a frame is placed in the Look Ahead Buffer (LAB) 58. These stages include some early Motion Estimation (ME) stages 54, Inverse Telecine stage 55 to transfer cinema signals to television, and the Group of Pictures (GOP) stage 56. The ME stage 54 is provided in addition to the ME stage information from the compressor of the encoder 41 and is used to determine the NP that helps the statmux controller 6 determine bandwidth need for the individual signal prior to encoding.
The output of the Pre-LAB 52 is provided to the Look Ahead Buffer (LAB) 58. The LAB 58 will buffer a fixed number of frames, for example a fixed 30 frames, regardless of the input format. With a fixed 30 frames, the clock timing of the LAB 58 will be different when 30 frames per second (fps) vs. a 60 fps output is desired.
The output of the LAB 58 is provided to the compressor and other components of the encoder 41. The final output of encoder 41 is then provided to multiplexer 8. The multiplexer 8 provides a Transport Stream (TS) Output Clock that clocks the output packets from the multiplexer 8. The TS output clock, as shown in
Other time references relative to the video input clock and the TS output clock are also illustrated in
The second state 82 is the “Bitrate Queue (BRQ) Driven and Need Parameter (NP) Sent” state. In state 82, the encoder state machine will transition to the BRQ driven state and start sending NP data to the controller once the encoder starts receiving bitrates. The encoder only sends NP data to the statmux controller when it is receiving BRQ data.
The third and final state 84 is the “Nominal Bitrate No NP” state. This nominal bitrate no NP state 84 is entered when a Hold command is sent by the statmux controller. The hold command is only used when the statmux controller is ceasing to function for any reason, such as when it is recovering from a failure. In the hold state all encoders in the system are sent to a nominal bitrate while the statmux controller is rebooted. No NP data should be sent by the encoders in this state. To prevent errors, the encoders should not transmit on the PCI bus while the controller is recovering.
Embodiments of the present invention provide for an improved determination of NP. The embodiments described take into account factors such as scene changes, and difficult frames that are identified in the video data provided for encoding.
A. Scene Change
After determining that a scene change will occur, the coded ratios stored in an encoder may not provide accurate determination for complexity that is provided as part of the NP data for the statmux controller. In the past, when determining a complexity the encoder looked only at the current picture and previous picture history. If a new scene is significantly more complex and requires a higher bit rate, the data complexity determination based only on current or previous data may not be adequate.
Discussion of a complexity determination will be made with respect to Appendix A, which includes C++ program code. The code of Appendix A is provided as an example to accomplish detection of a scene change and provide statistical information for the NP to enable controlling bit rate for an encoder. Also, reference will be made to
First in part A of the code of Appendix A and in step 90 of
Next in the code of Appendix A and in step 91 of
Next in Appendix A and step 92 in
If a scene change is detected, data within the new scene is specifically evaluated for complexity beginning in the code of section B of Appendix A. Initially for the new scene after the scene change, a best estimate is made for the coded ratios for the new scene. To do this, the code initially looks at the I, P and B type pictures. All I pictures from a scene tend to code similarly, and the same is true for P and B type pictures. However, the I, P and B type pictures can individually code very differently, so it is important to group complexity calculations by picture type. To ensure such a grouping, the code in section B determines on average what percentage of the pictures will be I, P and B type. These percentages are then used when determining the overall complexity.
Next, in a step of the code labeled “avgsScores, Pic_type counts, and difficult frame counts” calculations are made to determine complexity values for the current scene and the new scene using the average scores, picture type counts and difficult frame counts. Note from the code labeled “Do not include statistics from a previous scene” that only the new and current scene are evaluated. Data from the “previous scene,” where the previous scene is the scene immediately preceding the current scene is not included in the complexity determination.
Finally, in the code labeled “Using the standard GOP estimate if end of scene is not found or scene is longer than N” a limitation is made on the complexity evaluation. The limitation is based in part on the size of the LAB. If the entire scene does not fit within the LAB, the complexity determination is limited by the LAB size. Further if the length of the scene is longer than N, the maximum data that can be determined and provided to the statmux controller for the bit rate statistical analysis. N will be a limiting factor on the total complexity analysis.
The code in step C of Appendix A and in step 94 of
B. Difficult Frames
In one embodiment, the code further considers a particular class of “difficult frames.” Detecting difficult frames is also illustrated at step 95 in
With a determination of difficult frames as well as complexity due to scene changes, the code of step D of Appendix A and step 96 of
Although the present invention has been described above with particularity, this was merely to teach one of ordinary skill in the art how to make and use the invention. Many additional modifications will fall within the scope of the invention, as that scope is defined by the following claims.