Patent Application
20140321545
The present invention relates to the field of video encoding technologies, and in particular, to a video encoding method and device.
Generally, content of neighboring frames of a moving picture does not have a substantial change (except in a case such as scene switching), that is, a picture sequence has a strong temporal correlation. Therefore, the temporal correlation of a picture sequence may be utilized to use data of a previous frame picture as a reference frame to predict data of a next frame picture, so as to only transmit a difference between the data of the previous frame picture and the data of the next frame picture, thereby reducing an amount of data to be transmitted.
Reference frames are categorized to short-term reference frames and long-term reference frames. For a short-term reference frame, in a reference frame list, a first-in-first-out mechanism is adopted for the short-term reference frame, where, after a new picture is obtained by decoding, a previous short-term reference frame is removed from the reference picture list. A long-term reference frame is different from the short-term reference frame. The long-term reference frame exists in the reference picture list in a long term and serves as a reference for following pictures all the time until a specific syntax element is used to indicate that the long-term reference frame does not serve as a reference frame.
In a video encoding process, when a picture background does not change and picture content between neighboring frames moves, for example, an (n+1)th frame is a to-be-encoded current picture and an nth frame is a short-term reference picture of the current picture, for an area at a same location in the (n+1)th frame picture and the nth frame picture, if the area in the nth frame picture is picture content and the picture content is moving, the area at the same location in the (n+1)th frame picture becomes a picture background. If the nth frame is used as a short-term reference frame to predict and encode the (n+1)th frame picture, in the foregoing area, the picture background of the (n+1)th frame is predicted by referring to the picture content of the nth frame, resulting in a very large residual. Therefore, a coder selects an “intra (intra-frame)” mode to encode the (n+1)th frame. If a background frame is selected as a long-term reference frame and encoding is performed with reference to the long-term reference frame, and because content in the long-term reference frame is completely the same as a background of an area where a change exists between the (n+1)th frame and the nth frame, the coder selects a “skip” mode to encode a macroblock in the area. In comparison with the “intra” mode, the “skip” mode greatly saves a code rate. It can be seen that a compression efficiency of encoding with reference to the long-term reference frame is higher than that of encoding with reference to the short-term reference frame.
In the prior art, encoding with reference to a long-term reference frame generally has the following problems: When an irrecoverable error occurs at an encoding end or a decoding end, an Instantaneous Decoding Refresh (IDR) frame has to be sent to clear a reference picture buffer, which also clears the long-term reference frame which serves as a background frame, and as a result, the long-term reference frame which serves as the background frame is unavailable. In addition, if a current frame is incoherent with a neighboring frame in picture content, displaying the current frame by the decoding end causes a flicker in a displayed picture.
Embodiments of the present invention provide a video encoding method and device, so as to solve the problem that a long-term reference frame as a background frame is unavailable when an irrecoverable error occurs at an encoding end or a decoding end, and the problem that displaying a current frame by the decoding end causes a flicker in a displayed picture when the current frame is incoherent with a neighboring frame in picture content.
To achieve the foregoing objective, the embodiments of the present invention adopt the following technical solutions:
According to one aspect of the embodiments of the present invention, a video encoding method is provided, including: inputting a to-be-encoded video picture; encoding a current frame obtained according to the video picture and specifying the current frame as a long-term reference frame; encoding each frame of the video picture with reference to the long-term reference frame and sending a data packet obtained after the encoding to a decoding end; and after a refreshing frame request sent by the decoding end is received, if it is determined that the long-term reference frame still serves as a reference frame for encoding, sending a non-IDR frame picture to the decoding end, where a macroblock of the non-IDR frame picture includes a macroblock encoded with reference to the long-term reference frame, and/or a macroblock encoded with reference to an intra-frame macroblock.
According to another aspect of the embodiments of the present invention, a video encoding method is provided, including: inputting a to-be-encoded video picture; encoding a current frame obtained according to the video picture and specifying the current frame as a long-term reference frame; encoding each frame of the video picture with reference to the long-term reference frame and sending a data packet obtained after the encoding to a decoding end; and if it is determined that the current frame does not need to be displayed at the decoding end, indicating by means of encoding that the current frame is not to be displayed, so that the decoding end does not display the current frame after the video picture is received.
According to another aspect of the embodiments of the present invention, a video encoding device is provided, including: an input unit configured to input a to-be-encoded video picture; an encoding unit configured to encode a current frame obtained according to the video picture and specify the current frame as a long-term reference frame, encode each frame of the video picture with reference to the long-term reference frame, and send a data packet obtained after the encoding to a decoding end; a receiving unit configured to receive a refreshing frame request sent by the decoding end, and a refreshing frame sending unit configured to, after the refreshing frame request sent by the decoding end is received, if it is determined that the long-term reference frame still serves as a reference frame for encoding, send a non-IDR frame picture to the decoding end, where a macroblock of the non-IDR frame picture includes a macroblock encoded with reference to the long-term reference frame, and/or a macroblock encoded with reference to an intra-frame macroblock.
According to another aspect of the embodiments of the present invention, a video encoding device is provided, including: an input unit configured to input a to-be-encoded video picture; an encoding unit configured to encode a current frame obtained according to the video picture and specify the current frame as a long-term reference frame, encode each frame of the video picture with reference to the long-term reference frame, and send a data packet obtained after the encoding to a decoding end; and an indication unit configured to, if it is determined that the current frame does not need to be displayed at the decoding end, indicate by means of encoding that the current frame is not to be displayed, so that the decoding end does not display the current frame after the video picture is received.
In the video encoding method and device provided by the embodiments of the present invention, in a process of encoding with reference to a long-term reference frame, after a refreshing frame request sent by a decoding end is received, if it is determined that the long-term reference frame still serves as a reference frame, a picture encoded with reference to the long-term reference frame is sent to the decoding end, and/or a non-IDR frame picture encoded with reference to an intra-frame macroblock is sent to the decoding end. In this way, because a refreshing frame is not an IDR frame, not all buffered reference pictures are cleared; therefore, the long-term reference frame as a background frame is still available, thereby avoiding the problem that the long-term reference frame as the background frame is unavailable when an irrecoverable error occurs at an encoding end or the decoding end. In addition, when it is determined that a current frame does not need to be displayed, it is indicated by means of encoding that the current frame is not to be displayed, thereby avoiding the problem that displaying the current frame by the decoding end causes a flicker in a displayed picture when the current frame is incoherent with a neighboring frame in picture content.
To describe the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments. Apparently, the accompanying drawings in the following description show merely some embodiments of the present invention, and a person of ordinary skill in the art may still derive other drawings from these accompanying drawings without creative efforts.
The following clearly and describes the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are merely a part rather than all of the embodiments of the present invention. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
As shown in
S101: A video encoding device inputs a to-be-encoded video picture.
S102: The video encoding device encodes a current frame obtained according to the video picture and specifies the current frame as a long-term reference frame.
S103: The video encoding device encodes each frame of the video picture with reference to the long-term reference frame and sends a data packet obtained after the encoding to a decoding end.
S104: The video encoding device sends, after receiving a refreshing frame request sent by the decoding end, a non-IDR frame picture to the decoding end if it is determined that the long-term reference frame still serves as a reference frame for encoding, where a macroblock of the non-IDR frame picture includes a macroblock encoded with reference to the long-term reference frame, and/or a macroblock encoded with reference to an intra-frame macroblock.
In the video encoding method provided by the embodiment of the present invention, in a process of encoding with reference to a long-term reference frame, after a refreshing frame request sent by a decoding end is received, if it is determined that the long-term reference frame still serves as a reference frame, a picture encoded with reference to the long-term reference frame is sent to the decoding end, and/or a non-IDR frame picture encoded with reference to an intra-frame macroblock is sent to the decoding end. In this way, because a refreshing frame is not an IDR frame, not all buffered reference pictures are cleared; therefore, the long-term reference frame as a background frame is still available, thereby avoiding a problem that the long-term reference frame as the background frame is unavailable when an irrecoverable error occurs at an encoding end or the decoding end.
As shown in
S201: A video encoding device inputs a to-be-encoded video picture.
S202: The video encoding device encodes a current frame obtained according to the video picture and specifies the current frame as a long-term reference frame.
S203: The video encoding device encodes each frame of the video picture with reference to the long-term reference frame and sends a data packet obtained after the encoding to a decoding end.
S204: If determining that the current frame does not need to be displayed at the decoding end, the video encoding device indicates by means of encoding that the current frame is not to be displayed, so that the decoding end does not display the current frame after the video picture is received.
In the video encoding method provided by the embodiment of the present invention, when it is determined that a current frame does not need to be displayed, it is indicated by means of encoding that the current frame is not to be displayed, thereby avoiding a problem that displaying the current frame by a decoding end causes a flicker in a displayed picture when the current frame is incoherent with a neighboring frame in picture content.
As shown in
S301: A video encoding device inputs a to-be-encoded video picture.
S302: The video encoding device encodes a current frame obtained according to the video picture and specifies the current frame as a long-term reference frame.
Specifically, the video encoding device may obtain a frame of picture from the video picture as the long-term reference frame; or, synthesize at least two frames of picture from the video picture as the long-term reference frame; or, the video encoding device may photograph a background picture of a frame of picture in the video picture as the long-term reference frame; or, photograph background pictures of at least two frames of picture in the video picture and synthesize the background pictures of the at least two frames of picture as the long-term reference frame.
In addition, after the current frame is specified as the long-term reference frame, the video encoding device may further set a transmission priority for the long-term reference frame.
For example, in a scenario where a background basically does not change, the long-term reference picture as the background needs to serve as a reference in a long term; therefore, it is necessary to set a high transmission priority for the frame picture, and in a transmission process, correctness of code stream transmission of the frame is ensured preferentially.
S303: The video encoding device encodes each frame of the video picture with reference to the long-term reference frame and sends a data packet obtained after the encoding to a decoding end.
S304: The video encoding device sends, after receiving a refreshing frame request sent by the decoding end, a non-IDR frame picture to the decoding end if it is determined that the long-term reference frame still serves as a reference frame for encoding, where a macroblock of the non-IDR frame picture includes a macroblock encoded with reference to the long-term reference frame, and/or a macroblock encoded with reference to an intra-frame macroblock.
Specifically, when an irrecoverable error occurs at an encoding end or the decoding end, because the error occurs with encoding with reference to a current short-term reference frame, the short-term reference frame no longer serves as a reference and the decoding end sends the refreshing frame request to the video encoding device.
After the refreshing frame request sent by the decoding end is received, the video encoding device first determines whether all buffered reference pictures have to be cleared, and sends an IDR frame picture to the decoding end if it is determined that both the short-term reference frame and the long-term reference frame no longer serve as a reference frame, so as to clear all the buffered reference pictures including the short-term reference frame and the long-term reference frame. If it is determined that the long-term reference frame still serves as the reference frame, only the short-term reference frame needs to be cleared and a non-IDR frame picture is sent to the decoding end. A macroblock of the non-IDR frame picture includes a macroblock encoded with reference to the long-term reference frame, and/or a macroblock encoded with reference to an intra-frame macroblock. A picture following the non-IDR frame picture may be encoded with reference to the non-IDR frame picture rather than with reference to the short-term reference frame at which the error occurs. In this way, encoding is performed without reference to the short-term reference frame at which the error occurs, and a problem is avoided that the long-term reference frame as the background frame is cleared and is therefore unavailable.
S305: If determining that the current frame does not need to be displayed at the decoding end, the video encoding device indicates by means of encoding that the current frame is not to be displayed, so that the decoding end does not display the current frame after the video picture is received.
For example, if the current frame is a frame in the input to-be-encoded video picture, and the current frame is continuous and coherent with a neighboring frame in picture, the current frame picture should be output after decoding, so as to ensure a coherent and smooth decoded picture. If the current frame is a virtual picture synthesized according to multiple frames of to-be-encoded video picture, because the current frame picture is substantially different from a neighboring picture in content, and problems, such as an obvious flicker in an output picture, are caused if the current frame is selected and output, and therefore the current frame should not be output after decoding.
Specifically, when the current frame does not need to be used for display, the video encoding device may set, in the data packet sent to the decoding end, a predetermined flag indicating that the current frame is not to be displayed. In the existing H.264 protocol syntax, the video encoding device may appoint, in Supplemental Enhancement Information (SEI) packet content, a flag indicating that a code stream is not used for display after decoding, and a receiving end determines according to the flag that the current frame is not to be displayed after a picture is decoded.
The video encoding device may also indicate by setting a syntax element in the protocol syntax that the current frame is not to be displayed.
The video encoding device may also indicate by setting a field at a signaling transmission layer that the current frame is not to be displayed. For example, whether a code stream is used for display is indicated by a Contributing source (CSRC) field in a Real-time Transport Protocol (RTP) packet.
In the video encoding method provided by the embodiment of the present invention, in a process of encoding with reference to a long-term reference frame, after a refreshing frame request sent by a decoding end is received, if it is determined that the long-term reference frame still serves as a reference frame, a picture encoded with reference to the long-term reference frame is sent to the decoding end, and/or a non-IDR frame picture encoded with reference to an intra-frame macroblock is sent to the decoding end. In this way, because a refreshing frame is not an IDR frame, not all buffered reference pictures are cleared; therefore, the long-term reference frame as a background frame is still available, thereby avoiding the problem that the long-term reference frame as the background frame is unavailable when an irrecoverable error occurs at an encoding end or the decoding end. In addition, when it is determined that the current long-term reference frame does not need to be displayed, it is indicated by means of encoding that the current long-term reference frame is not to be displayed, thereby avoiding a problem that displaying the current frame by the decoding end causes a flicker in a displayed picture when the current frame is incoherent with a neighboring frame in picture content.
As shown in
S401: A video encoding device inputs a to-be-encoded video picture.
S402: The video encoding device encodes a current frame obtained according to the video picture.
S403: The video encoding device determines whether it is needed to clear a long-term reference frame after a refreshing frame request sent by a decoding end is received; if needed, execute step S404; and if not needed, execute step S405.
S404: The video encoding device instructs that the current frame be encoded as an IDR frame picture, so as to clear all buffered reference pictures.
For example, the video encoding device may encode all macroblocks in a frame of picture by using an intra-frame compression mode and set a Nal_unit_type to 5 to obtain an IDR frame by encoding, so as to clear all the buffered reference pictures by using the IDR picture as a refreshing frame.
S405: The video encoding device instructs that the current frame be encoded as a non-IDR frame picture, where a macroblock of the non-IDR frame picture includes a macroblock encoded with reference to the long-term reference frame, and/or a macroblock encoded with reference to an intra-frame macroblock.
For example, the video encoding device may implement a refreshing frame by encoding all macroblocks in a picture by using an intra-frame compression mode or by using an inter-frame prediction mode with reference to the long-term reference frame, so as to ensure correctness of decoding performed by a receiving end. Moreover, a nal_unit_type is set to a value other than 5 so as to obtain a non-IDR frame by encoding, so that the long-term reference frame in a reference picture buffer can continue to serve as a reference.
S406: The video encoding device determines whether a current picture is a long-term reference frame; if yes, execute step S407; and if not, execute step S408.
For example, in a encoding process, it may be stipulated that, during picture encoding, a frame is specified or a frame of picture is inserted every 200 frames to serve as a long-term reference frame; and the video encoding device may determine whether the current frame is a long-term reference frame according to the counted number of frames.
S407: The video encoding device marks the current picture as a long-term reference frame.
Specifically, the video encoding device may mark the current picture as the long-term reference frame by separately setting an adaptive_ref_pic_marking_mode_flag flag to 1, setting a memory_management_control_operation flag to 6, and setting a long_term_frame_idx value of the current frame.
S408: The video encoding device does not mark the current picture as a long-term reference frame and marks a short-term reference frame as a long-term reference frame.
Specifically, the video encoding device may not mark the current picture as the long-term reference frame by separately setting the adaptive_ref_pic_marking_mode_flag flag to 1, setting the memory_management_cintrol_operation flag to 3, and setting a long_term_frame_idx value of a short-term reference frame, so as to mark the short-term reference frame as the long-term reference frame.
S409: The video encoding device sets a transmission priority of the long-term reference frame.
S410: The video encoding device determines whether the current frame needs to be displayed; when it is determined that the current frame does not need to be displayed, execute step S411; and when it is determined that the current frame needs to be used for displaying, execute step S412.
S411: The video encoding device sets a no display flag.
Specifically, in the H.264 protocol, whether full-frame freeze or full-frame freeze release are performed is specified by using Full-frame freeze SEI or Full-frame freeze release SEI. When full-frame freeze is performed, decoding of the current picture is completed, and the current picture does not need to update displayed video frame content, so as to keep the displayed video frame content unchanged, thereby achieving a no display purpose after a single frame or multiple frames of picture are decoded.
For example, when it is only needed that the long-term reference frame not be displayed in a current frame, it may be implemented by sending Full-frame freeze SEI (SEI payloadtype=13) and setting a full_frame_freeze_repetition_period to 0.
For another example, when it is needed that multiple frames of picture not be displayed, the Full-frame freeze SEI (SEI payloadtype=13) is sent in a starting frame of long-term reference frames that do not need to be displayed, and the full_frame_freeze_repetition_period is set to 1. A piece of full-frame freeze release SEI (SEI payloadtype=14) is sent in a frame where freeze release is needed, so as to implement that the multiple frames of picture are not displayed.
For still another example, when it is needed that multiple frames of picture not be displayed, the Full-frame freeze SEI (SEI payloadtype=13) is sent in a starting frame of long-term reference frames that do not need to be displayed, and the full_frame_freeze_repetition_period is set to 1. An sequence parameter set (SPS) or a picture parameter set (PPS) is sent in a frame where freeze release is needed, and a new IDR frame is obtained by encoding to start a new sequence, so as to perform the freeze release automatically.
S412: The video encoding device sets a display flag.
S413: The video encoding device outputs a code stream of the encoded video picture.
In the video encoding method provided by the embodiment of the present invention, in a process of encoding with reference to a long-term reference frame, after a refreshing frame request sent by a decoding end is received, if it is determined that the long-term reference frame still serves as a reference frame, a picture encoded with reference to the long-term reference frame is sent to the decoding end, and/or a non-IDR frame picture encoded with reference to an intra-frame macroblock is sent to the decoding end. In this way, because a refreshing frame is not an IDR frame, the existing long-term reference frame is not cleared; thereby avoiding a problem that the long-term reference frame as the background frame is unavailable when an irrecoverable error occurs at an encoding end or the decoding end. In addition, when it is determined that a current frame does not need to be displayed, it is indicated by means of encoding that the current frame is not to be displayed, thereby avoiding a problem that displaying the current frame by the decoding end causes a flicker in a displayed picture when the current frame is incoherent with a neighboring frame in picture content.
A video encoding device 50 provided by an embodiment of the present invention corresponds to the foregoing method embodiments, and can be used for all steps in the foregoing method embodiments. Specific steps of the video encoding device 50 are described in the foregoing method embodiments and are not described in detail again herein. As shown in
The video encoding device provided by the embodiment of the present invention sends, in a process of encoding with reference to a long-term reference frame, after a refreshing frame request sent by a decoding end is received, a picture encoded with reference to the long-term reference frame to the decoding end, and/or a non-IDR frame picture encoded with reference to an intra-frame macroblock to the decoding end if it is determined that the long-term reference frame still serves as a reference frame. In this way, because a refreshing frame is not an IDR frame, not all buffered reference pictures are cleared; therefore, the long-term reference frame as a background frame is still available, thereby avoiding a problem that the long-term reference frame as the background frame is unavailable when an irrecoverable error occurs at an encoding end or the decoding end.
Further, as shown in
The refreshing frame sending unit 504 may be further configured to, after the refreshing frame request sent by the decoding end is received, if it is determined that the long-term reference frame does not serve as a reference frame for encoding, send an IDR frame picture to the decoding end.
A video encoding device 70 provided by an embodiment of the present invention corresponds to the foregoing method embodiments, and can be used for all steps in the foregoing method embodiments. Specific steps of the video encoding device 70 are described in the foregoing method embodiments and are not described in detail again herein. As shown in
The video encoding device provided by the embodiment of the present invention indicates by means of encoding that a current frame is not to be displayed when it is determined that the current frame does not need to be displayed, thereby avoiding a problem that displaying the current frame by a decoding end causes a flicker in a displayed picture when the current frame is incoherent with a neighboring frame in picture content.
A person of ordinary skill in the art may understand that all or a part of the steps of the method embodiments may be implemented by a program instructing relevant hardware. The program may be stored in a computer readable storage medium. When the program runs, the steps of the method embodiments are performed. The foregoing storage medium includes any medium that can store program code, such as a read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.
The foregoing descriptions are merely specific embodiments of the present invention, but are not intended to limit the protection scope of the present invention. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in the present invention shall fall within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201110427151.1 | Dec 2011 | CN | national |
This application is a continuation of International Application No. PCT/CN2012/080438, filed on Aug. 22, 2012, which claims priority to Chinese Patent Application No. 201110427151.1, filed on Dec. 19, 2011, both of which are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2012/080438 | Aug 2012 | US |
| Child | 14308791 | US |
