The present application claims priority from Japanese application JP2006-306135 filed on Nov. 13, 2006, the content of which is hereby incorporated by reference into this application.
1. Field of the Invention
The present invention relates to a technology for encoding a moving image.
2. Description of the Related Art
In the encoding technology in which predictive images are generated from reference images for encoding a moving image, a technology, such as the one disclosed in US2006/0171680 (corresponding to JP-A-2006-217180), is known for determining whether or not an image is to be deleted from the reference image memory.
However, when inter-screen prediction is performed for a slow-moving video or a video with little movement in the prior art, always using the temporally closest image as the reference frame of the video generates a problem that the reference frame update processing involves an amount of calculation that is redundant.
On the other hand, using a temporally distant image as the reference frame of a fast-moving video or a video with large movement generates a problem that the encoding efficiency is significantly decreased.
In view of the fore going, the present invention reduces the amount of calculation processing in the encoding processing while suppressing a decrease in encoding efficiency.
One embodiment of the present invention comprise the steps of generating predicted image data from an image of a predetermined reference frame; generating differential image data from a difference between the predicted image data and image data of one frame of the input image data; performing discrete cosine transformation processing and quantization processing for the differential image data for generating encoded image data; performing variable-length encoding processing for the encoded image data for generating an encoded stream; and performing reference frame update determination processing in which a correlation between the image of the reference frame and the image of the one frame is determined for deciding whether or not the one frame is to be used as a new reference frame.
The above configuration controls whether or not the reference frame update processing is to be performed and reduces the amount of calculation processing during the encoding processing while suppressing a decrease in encoding efficiency.
The present invention can reduce the amount of calculation processing during the encoding processing while suppressing a decrease in encoding efficiency.
These and other features, objects and advantages of the present invention will become more apparent from the following description when taken in conjunction with the accompanying drawings wherein:
While we have shown and described several embodiments in accordance with our invention, it should be understood that disclosed embodiments are susceptible of changes and modifications without departing from the scope of the invention. Therefore, we do not intent to be bound by the details shown and described herein but intend to cover all such changes and modifications a fall within the ambit of the appended claims.
Embodiments of the present invention will be described below with reference to the drawings.
In all of the drawings, the same reference numeral is given to components having the same function.
The expression “update processing” or “reference frame update processing” used in the description and the drawings of this specification refers to the processing for storing, holding, or saving target image data in the memory or the storage unit, in which reference image data is held, as reference image data. In this case, the reference image data, which has been held in the memory or the storage unit, may or may not be discarded or erased.
An I-picture used in the description and the drawings of this specification refers to a frame encoded with no reference to other frames.
A P-picture used in the description and the drawings of this specification refers to a frame encoded with reference to a frame that is a past frame of the encoding frame. That is, a P-picture is a frame encoded through forward prediction.
A B-picture used in the description and the drawings of this specification refers to a frame encoded with reference to a frame that is a past frame and a future frame of the encoding frame. That is, a B-picture is a frame encoded through bi-directional prediction.
A frame simply called a “next frame” in the description and the drawings of this specification refers to a temporally immediately following frame.
First, the following describes a first embodiment of the present invention with reference to the drawings.
Referring to frames (501)-(507) in
To solve this problem, the method in this embodiment determines whether or not update processing is necessary for a reference frame according to the characteristics of a moving image, and reduces the calculation processing amount during the encoding processing while suppressing a decrease in encoding efficiency.
The following describes this method with the frames (508)-(514) in
The processing described above reduces the amount of reference frame update processing and, so, reduces the amount of processing required for encoding. For example, in the encoding processing in the prior art shown in
The method described above uses the correlation between a reference frame and an encoding frame to determine whether the reference frame update processing is necessary, thus reducing the number of times the update processing is performed.
When an encoded stream is generated using one example of the encoding processing method shown in
When an encoded stream is generated using one example of the encoding processing method shown in
At this time, when the frame (for example, frame (512) in
The correlation determination method will be described in detail later.
Next, the following describes an example of a moving image encoding apparatus in this embodiment with reference to
For example, a moving image encoding apparatus (100) comprises an input image memory (101) in which input image data is held, a predicted image generation unit (102) that performs intra-screen prediction or inter-screen prediction for input image data, one block at a time, for generating predicted image data, a subtracter (109) that calculates the difference between predicted image data and input image data to generate differential image data, an encoding processing unit (103) that frequency-converts, quantizes, and encodes differential image data, a variable-length encoding unit (104) that efficiently encodes image data based on the generation probability of symbols, a reference frame update determination processing unit (105) that determines whether or not a reference frame is updated, a decoding processing unit (106) that de-quantizes and inverse-orthogonal transforms encoded differential image data for decoding, an adder (110) that combines decoded differential image data with predicted image data to generate reference image data, a reference image memory (107) that holds generated reference image data, a switch unit (108) that connects the reference frame update determination processing unit (105) and the decoding processing unit (106), and a control unit (150) that controls the components of the moving image encoding apparatus (100). A block is a small area generated by dividing an image. The variable-length encoding unit (104) is though of as an output unit that outputs an encoded stream.
The input image memory (101) holds input image data and sends it to the predicted image generation unit (102). The predicted image generation unit (102) divides the input image into blocks of predetermined size and selects an encoding mode, which maximizes the prediction efficiency, for each block from the pre-set encoding modes. That is, the predicted image generation unit (102) processes a part of the input image. Next, the predicted image generation unit (102) generates predicted image data in the selected encoding mode. An encoding mode refers to a combination of encoding methods, for example, the prediction method, block size, and pixel scan method, that can be switched from block to block.
Depending upon the encoding mode used, the predicted image generation unit (102) may acquire the reference image data, or a part, of a reference frame held in the reference image memory (107) and, using the acquired data, generates predicted image data. The predicted image generation unit (102) sends the generated predicted image data to the subtracter (109) and the adder (110).
Next, the subtracter (109) calculates the difference between the image data of the input image data or its part and the predicted image data on a pixel basis and generates differential image data. The subtracter (109) sends the generated differential image data to the encoding processing unit (103) and the reference frame update determination processing unit (105). The encoding processing unit (103) performs DCT (Discrete Cosine Transformation) processing and quantization processing for the acquired differential image data. The encoding processing unit (103) also sends the processed encoded image data to the variable-length encoding processing unit (104), reference frame update determination processing unit (105), and switch unit (108). In addition, the variable-length encoding processing unit (104) variable-length encodes the encoded image data based on the generation probability of symbols to generate an encoded stream and outputs the generated encoded stream outside the moving image encoding apparatus 100. The variable-length encoding processing unit (104) sends the encoded stream also to the reference frame update determination processing unit (105). The reference frame update determination processing unit (105) determines the picture type of the encoding frame from the acquired differential image data, encoded image data, or encoded stream. If the picture type is a P-picture, the reference frame update determination processing unit (105) calculates the prediction error value or the generation code amount of the encoding image. And, using the predetermined fixed determination criteria, statistical determination criteria, or local determination criteria, the reference frame update determination processing unit (105) determines the correlation between the encoding image and the reference frame. Based on the determined picture type and the correlation determination result, the reference frame update determination processing unit (105) sends the switch open/close control signal to the switch unit (108). At this time, if the picture type is an I-picture and if the picture type is a P-picture and the correlation is determined to be low, the reference frame update determination processing unit (105) sends the switch close signal. If the picture type is a P-picture and the correlation is determined to be high, the reference frame update determination processing unit (105) sends the switch open signal.
When the switch close signal is sent from the reference frame update determination processing unit (105), the switch unit (108) closes the switch to send the encoded image data to the decoding processing unit (106). The decoding processing unit (106) performs the inverse quantization processing and IDCT (Inverse DCT: Inverse Discrete Cosine Transformation) processing for the acquired encoded image data or the encoded stream, one block at a time, to decode it to the differential image data, and sends it to the adder (110). Next, the adder (110) combines the differential image data acquired from the decoding processing unit (106) and the predicted image data acquired from the predicted image generation unit (102) to generate the image data of the reference image frame. Next, the adder (110) sends the reference image data to the reference image memory (107) and the reference image memory (107) stores the reference image data. The stored reference image data is used for the predicted image generation processing of the predicted image generation unit (102) as necessary.
When the switch open signal is sent from the reference frame update determination processing unit (105), the switch unit (108) opens the switch to prevent the encoded image data from being sent to the decoding processing unit (106). Therefore, no processing is performed thereafter.
In the description of the moving image encoding apparatus 100 in
The moving image encoding apparatus 100 described above can provide a moving image encoding apparatus that selects whether or not reference image data is generated and stored according to the correlation between encoding image and a reference frame.
The picture type determination unit (210) determines the picture type of a target encoding frame input to the reference frame update determination processing unit (105) based on the picture type information acquired from the control unit (150). If the picture type of the target encoding frame is an I-picture, the picture type determination unit (210) sends the switch close signal to the switch unit (108) as the switch open/close control signal. If the picture type of the target encoding frame is a P-picture, the picture type determination unit (210) sends the differential image data to the prediction error value calculation unit (201) and sends the encoded image data or encoded stream to the generation code amount calculation unit (202). The picture type information acquired from the control unit (150) is determined by the control unit (150) from the memory address of the start point or end point of the encoded image data or the encoded stream.
Next, the prediction error value calculation unit (201) calculates the size of each component of the differential image data input from the subtracter (109) and, for each frame, calculates the total value as the prediction error value. The generation code amount calculation unit (202) calculates the amount of code generated when an encoding frame is encoded, using the encoded stream acquired from the variable-length encoding processing unit (104) or the encoding image data acquired from the encoding processing unit (103).
In addition, the update determination processing unit (203) determines the correlation between the target encoding image and the reference frame by using one of the following three types of determination criterion information: fixed determination criterion information which is predefined and is stored in the fixed determination criterion memory (209), local determination criterion information stored in the local determination criterion memory (205), and statistical determination criterion information stored in the statistical determination criterion memory (204).
The following describes a determination method that uses the fixed determination criterion information, a determination method that uses the local determination criterion information, and a determination method that uses the statistical determination criterion information.
First, as an example of a first determination method, the following describes the determination method that uses the fixed determination criterion information. In the first determination method, the parameter of the target encoding frame Yij and predetermined data that is predefined are compared to determine if the reference frame is to be updated. For example, the following expression, expression 1, is used for the determination.
SAD(Yij)≦α or FB(Yij)≦β Expression 1
where SAD(Yij) is the prediction error sum of the target encoding frame Yij. The prediction error sum is the sum of prediction errors in the blocks in an image calculated for each frame. The prediction errors are calculated, for example, by the prediction error value calculation unit (201) in
The fixed determination criterion information used in the example of this determination method is information such as α and β. The fixed determination criterion information should be pre-set by evaluating the encoding efficiency and saved in the fixed determination criterion memory (209).
The value on the right-hand side of expression 1 is called the determination criterion value of the fixed determination criterion, and the value on the left-hand side is called the determination target calculation value.
In this determination method, satisfying expression 1 means that the correlation between the reference frame Xi and the encoding frame Yij is high. If SAD(Yij) or FB(Yij) exceeds a predetermined amount and does not satisfy expression 1, the correlation is determined to be low.
For example, when the first determination method is used, the update determination processing unit (203) of the reference frame update determination processing unit (105) shown in
If it is determined that expression 1 is not satisfied, the update determination processing unit (203) judges that the correlation between the reference frame Xi and the target encoding frame Yij is low. In this case, the update determination processing unit (203) sends the switch close signal to the switch unit (108). In response to this signal, the switch unit (108) sends the encoded image data of the encoding frame Yij, sent from the encoding processing unit (103), to the decoding processing unit (106). The encoded image data of the encoding frame Yij is added to the predicted image data of the encoding frame Yij by the adder (110). The added-up data is saved in the reference image memory (107) as new reference image data and then the reference frame update processing is completed. As a result, the target encoding frame Yij becomes a new reference frame Xk(where, k=i+1).
If it is determined that expression 1 is satisfied, the update determination processing unit (203) sends the switch open signal to the switch unit (108). In response to this signal, the switch unit (108) opens the switch to prevent the reference frame from being updated and, in this way, reduces the number of times the update processing is performed.
The determination processing, in which fixed determination criterion information is used as described above, updates the reference frame if the correlation between the encoding frame and the current reference frame is judged lower than a predetermined criterion that is predefined.
For example, in a frame where the scene is switched to a low-correlation scene or where the input image is switched from a slow-moving scene or a scene with little movement to a fast-moving scene or a scene with large movement, the value of the prediction error sum SAD(Yij) of the target encoding frame Yij or the generation code amount FB(Yij) of the encoding frame Yij becomes large. Setting α or β in advance so that expression 1 is not satisfied in such a case, the reference frame is updated. After that, if the input image is switched to a slow-moving scene or a scene with little movement, SAD(Yij) or FB(Yij) does not exceed α or β and, so, the reference frame update processing is suppressed. This reduces the amount of calculation processing involved in the update processing. In addition, the determination processing using the fixed determination criterion information limits an increase in the generation code amount, thus preventing the coding efficiency from being decreased. Therefore, the reference frame update processing using the fixed determination criterion information reduces the amount of calculation of the encoding processing while suppressing a decrease in encoding efficiency.
Next, as an example of a second determination method, the following describes the determination method that uses the local determination criterion information. In the second determination method, the parameter of the target encoding frame Yij is compared with the parameter of the frame Yil, which is the next frame of the reference frame Xi, for determining whether the reference frame is to be updated.
In one example of the determination processing using the local determination criterion information, expression 2 given below is used for the determination.
SAD(Yij)≦SAD(Yil)×y or FB(Yij)≦FB(Yil)×δ Expression 2
where, SAD(Yij) is the prediction error sum of the target encoding frame Yij, and SAD(Yil) is the prediction error sum of the frame Yil that is the next frame of the reference frame Xi. FB(Yij) is the generation code amount of the target encoding frame Yij, and FB(Yil) is the generation code amount of the frame Yil that is the next frame of the reference frame Xi. γ and δ are constants.
The local determination criterion information used in this embodiment is, for example, information such as SAD(Yil), FB(Yil), γ, and δ. That is, the local determination criterion information refers to the parameters other than those (SAD(Yij) and FB(Yij) in expression 2) for the encoding frame in expression 2.
The right-hand side of expression 2 is called the determination criterion value of the local determination criterion, and the left-hand side is called the determination target calculation value.
In the reference frame update determination processing unit (105) in
If the update determination processing unit (203) determines that expression 2 is not satisfied, that is, if it is determined that the correlation between the reference frame Xi and the target encoding frame Yij does not satisfy the local determination criterion, the reference frame update processing is the same as that performed when it is determined that expression 1 is not satisfied. As a result, the data of the new reference image Xk (where k=i+1) is saved in the reference image memory (107).
When the current reference image is changed to the new reference image Xk, the prediction error sum or the generation code amount of the immediately-following target encoding frame Ykl becomes one of new local determination criterion information. Therefore, the local determination criterion update unit (207) acquires the prediction error sum or the generation code amount of the frame Ykl from the update determination processing unit (203) and stores it in the local determination criterion memory (205). The prediction error sum or the generation code amount of the frame Ykl is used in the subsequent determination processing.
If it is determined that expression 2 is satisfied, the update determination processing unit (203) performs the same processing as when it is determined that expression 1 is satisfied. In this case, the reference frame is not updated and, so, the number of times the update processing is performed can be reduced.
When the determination processing using the local determination criterion information as described above is used, the reference frame is updated if the correlation between the encoding frame and the current reference frame gets worse than the correlation between the current reference frame and the frame immediately after the reference frame.
For example, when the reference frame updating is suppressed in a scene where the input image is a slow-moving image, the correlation becomes lower as the target encoding frame and the current reference frame become temporally distant. That is, the value of the prediction error sum SAD(Yij) of the target encoding frame Yij or the value of the generation code amount FB(Yij) of the target encoding frame Yij gradually gets larger. This determination method can determine whether or not the reference frame should be updated based on the parameter of the frame Yil that is the next frame of the reference frame Xi.
As long as the correlation satisfies the local determination criterion based on the next frame of the reference frame, the update processing of the reference frame is suppressed. The reference frame is updated when the correlation does not satisfy the local determination criterion. Therefore, the reference frame update determination processing using the local determination criterion information can reduce the calculation processing amount of the encoding processing while suppressing a decrease in encoding efficiency.
Unlike the first determination method, the second determination method does not use the fixed determination criterion, allowing the correlation to be determined according to the input image. For example, in the case where the effective absolute values such as the fixed determination criterion information (a or can not be easily decided in advance, the second determination method is used. That is, the relative local determination criterion is set based on the parameter of the next frame of the reference frame.
Although the parameter of the frame Yil is multiplied by a constant on the right-hand side of expression 2, this is only exemplary. Instead, a function that has the frame Yil parameter as a variable may be on the right-hand side. For example, the average value may be calculated, or the weighted-sum may be calculated, with the Yil parameter as a variable. The average, median, or most-frequent value, statistically calculated from the distribution of multiple the Yil parameter, may be used. In either case, the determination processing using expression 2 achieves the effect that the calculation processing amount of the encoding processing is reduced while suppressing a decrease in encoding efficiency.
Next, as an example of a third determination method, the following describes the determination method that uses the statistical determination criterion information. In the third determination method, the data that is compared with the parameter of the encoding frame Yij is data calculated from the parameters of multiple frames. One example of the data that is compared with the parameter of the encoding frame Yij is the parameters of the frames Yml immediately after Xm(m<i) that is a reference frame (past reference frame) temporally preceding the reference frame Xi of the encoding frame Yij. That is, in the determination processing using the statistical determination criterion information in this embodiment, the statistical function, which uses the parameters of multiple frames Yal-Ybl as variables with m varying from a to b (a≦b≦i), is used as the statistical determination criterion information.
In one example of determination processing using the statistical determination criterion information, expression 3 given below is used for the determination.
SAD(Yij)≦
where SAD(Yij) and FB(Yij) are the same parameters of a target encoding image as those in expression 1 and expression 2. For example,
The statistical determination criterion information used in this embodiment is, for example, information such as
The right-hand side of expression 3 is called the determination criterion value of the statistical determination criterion, and the left-hand side is called the determination target calculation value.
In the reference frame update determination processing unit (105) in
If the update determination processing unit (203) determines that expression 3 is not satisfied and that the parameter of the encoding frame Yij does not satisfy the statistical determination criterion, the reference frame update processing is the same as that performed when it is determined that expression 1 is not satisfied. As a result, the data of the new reference image Xk (where k=i+1) is saved in the reference image memory (107).
If the update determination processing unit (203) determines that expression 3 is satisfied, the processing is the same as that performed when it is determined that expression 1 is satisfied. The reference frame is not updated and, so, the number of times the update processing is performed is reduced.
The parameters of past reference images are used for the statistical determination criterion information. Thus, the parameters (for example, prediction error sum or generation code amount) of the encoding frame Ykl, which follows the new reference image Xk, are also one of the parameters used for the subsequent statistical determination criterion information. Therefore, the statistical determination criterion update unit (206) acquires the prediction error sum or the generation code amount of the frame Ykl from the update determination processing unit (203) and stores it in the statistical determination criterion memory (204). The prediction error sum or the generation code amount of the frame Ykl is used in the subsequent determination processing.
The following describes the advantages of using the statistical determination criterion information in the reference frame update determination processing.
The first determination method was described before in which the reference frame update processing is performed using the fixed determination criterion information that is predefined. However, the proper value of the fixed determination criterion information, α or β, differs according to the type of an input image. In this case, the third determination method allows the determination criterion of the reference frame update processing to be changed according to the parameters of multiple frames Yml that continue to the encoding frame. That is, the determination criterion of the reference frame update processing can be determined according to the motion of the image of an input image in the scene for a predetermined period that continues to the encoding frame. For example, when the input image is switched from a slow-moving scene to a fast-moving scene, the prediction error sum or the generation code amount in each frame after the input image is switched to the fast-moving scene becomes larger than that of the scene where the input image moves slowly. The third determination method, if used in this case, allows the determination criterion of the reference frame update processing to be defined based on the scene in which the input image moves slowly.
So, even on a device that receives an input image that cannot be predicted in advance, this method compares the amount of the motion of an input image among scenes when the input image is switched from one scene to another and, based on the comparison result, properly determines whether the reference frame should be updated.
The second determination method can also determine the determination criterion of the update processing of a reference frame according to an input image. In the second determination method, assume that, when the input image is switched from a slow-moving scene to a fast-moving scene, the reference frame is updated when the input image is switched to the fast-moving scene. After that, if the fast-moving scene appears consecutively, the correlation between the consecutive frames remains low because the input image moves fast. In this case, to reduce the generation code amount and to keep good coding efficiency, it is desirable that the reference frame be updated frequently. However, even in a case where the generation code amount is large, expression 2 is satisfied in some cases if the amount specified by the parameter of the target encoding frame is almost the same as that of the parameter of the frame Yil that is the next frame of the reference frame Xi. Because the generation code amount is large in this case, the reference frame should preferably be updated for reducing the generation code amount and for achieving better encoding efficiency. Therefore, in such a case, the second determination method and the third determination method should be combined. This combination allows the fast-moving scene of the input image to be determined based on the parameter of the slow-moving scene and, as a result, increases the number of reference frame updates and properly reduces the generation code amount.
In the third method, the fast-moving scene of an input image is determined primarily based on the parameters of a slow-moving scene to reduce the calculation processing amount of the encoding processing while suppressing a decrease in encoding efficiency. Therefore, when the frames Yal-Ybl are selected for calculating the parameters used for expression 3, only those past frames Yml(m<i) may be selected whose parameter (prediction error sum or generation code amount) is equal to or smaller than a predetermined value that is predefined. Alternatively, the multiple frames may be selected whose parameter (prediction error sum or generation code amount) is equal to or less than a predetermined value and which consecutively appear a specified number of times or more. Doing so advantageously gives the average value of the parameters of slow-moving scenes.
Although the parameter of the multiple frames Yal-Ybl is multiplied by a constant on the right-hand side of expression 3, this is only exemplary. Instead, a function that has the parameters of multiple frames Yal-Ybl as variables may be on the right-hand side. For example, the average value may be calculated, or the weighted-sum may be calculated, with the Yil parameters as variables. The average, median, or most-frequent value, statistically calculated from the distribution of multiple Yil parameters, may also be used. In either case, the determination processing using expression 3 achieves the effect that the calculation processing amount of the encoding processing is reduced while suppressing a decrease in encoding efficiency.
Three determination methods have been described. In addition to the prediction error value or the generation code amount, the following may be used for the parameter in the second determination method and the third determination method: the motion vector size, the code amount of the motion vector, the quantization error value, the ratio of intra-predicted blocks in a frame, or the prediction error value for which frequency conversion such as Hadamard transform, is performed. That is, any parameter related to the encoding efficiency or the prediction efficiency of a target encoding image may be used.
In the above description, though the parameter is the sum in a frame, a value calculated with a weight applied based on the position or the type of a block may also be used. Therefore, the parameter may be selected based on the position or the type of a block.
The three methods described above may be used singly or in combination as one determination method in this embodiment.
Next,
First, encoding processing is performed for an encoding frame (701). The processing (701) is performed by the predicted image generation unit (102), subtracter (109), encoding processing unit (103), and variable-length encoding processing unit (104) described in the operation of the moving image encoding apparatus (100) in
On the other hand, if the encoding frame is determined to be a P picture in the determination step (702), the reference frame update determination processing unit (105) uses the determination method of this embodiment to determine if the reference frame is to be updated (705). If the reference frame update determination processing unit (105) decides that the reference frame must be updated, the reference frame is updated (703) and control is passed to the determination step (706). Conversely, if the reference frame update determination processing unit (105) decides that the reference frame need not be updated, the reference frame update determination processing unit (105) sends the switch open signal to the switch unit (108). This prevents the reference frame from being updated and passes control to the determination step (706).
In the determination step (706), a determination is made whether or not the encoding processing has been completed for all frames. This determination is made, for example, by the control unit (150). If the encoding processing is not yet completed, the encoding processing is performed for the next frame (701). On the other hand, if the encoding processing is completed, the processing is terminated.
The operation of the update determination processing unit and the moving image encoding apparatus in the first embodiment described above or the operation of the components in the flow of the encoding processing performed by them may be implemented by the autonomous operation of the components or by the instruction from the control unit (150). The control unit (150) and the software may work together to implement the operation.
The update determination processing unit and the moving image encoding apparatus in the first embodiment described above or the encoding processing performed by them controls whether or not a reference frame is to be updated. This control operation reduces the number of times the reference frame update processing is performed in a slow-moving scene and reduces the generation amount of calculation processing associated with the reference frame update processing such as the decoding processing, memory transfer processing, or other processing using decoded image data. The generation amount of calculation processing can be reduced while suppressing a decrease in encoding efficiency.
That is, the first embodiment of the present invention reduces the amount of calculation processing during the encoding processing while suppressing a decrease in encoding efficiency.
Next, the following describes a second embodiment of the present invention with reference to the drawings.
Referring to the frames (601)-(607) in
In this case, the reference frame must be updated after the encoding processing of all I P-pictures or P-pictures. Therefore, the update processing is performed frequently and, as a result, the processing amount of the update processing is large.
To solve this problem, the second embodiment uses not only the update processing method described in the first embodiment but also a method, which changes the picture structure according to the property of a moving image, to reduce the calculation processing amount during the encoding processing while suppressing a decrease in encoding efficiency.
The following describes this method with reference to
In the encoding method in this embodiment, the picture type of each frame Zij is determined according to the correlation between frame Vi, which temporally precedes Zij, and frame Vn(n=i+1) which temporally follows.
When frame Vn is encoded, each frame Zij is not yet encoded and, so, this frame Zij is thought of as an un-encoded frame arranged temporally between frame Vi and frame Vn at this time.
At this time, if the correlation between frame Vi and frame Vn is low, the picture type of each frame Zij is a B-picture. In this case, as in the prior-art technology shown in
If the correlation between frame Vi and frame Vn is low, the picture type of each frame Zij is a B-picture because of the following reason. That is, when a frame between frame Vi and frame Vn is encoded, the correlation between a target encoding frame arranged between both frames and frame Vi gets lower as the encoding frame gets nearer to frame Vn. In the forward prediction encoding in which frame Vi is used as, the reference frame, a predictive image is generated based on frame Vi. So, as the target encoding frame gets nearer to frame Vn, the prediction error value and the generation code amount become large. In contrast, if bi-directional prediction encoding in which the frame Vi and the frame Vn are used as the reference frames, the predictive image can be generated by selecting one of the two types of encoding, that is, the encoding using frame Vi and frame Vn and the encoding using one of both frames, whichever is better in encoding efficiency. Thus, this method can suppress the prediction error value and the generation code amount of an encoding frame.
Next, if the correlation between frame Vi and frame Vn is high, the picture type of each frame Zij is a P-picture. In the description of this embodiment, data arranged in such a picture type order is represented as the expression “the picture structure is the PP structure.”
If the correlation between frame Vi and frame Vn is high, the picture type of each frame Zij is a P-picture because of the following reason. That is, when a frame between frame Vi and frame Vn is encoded, the correlation between a target encoding frame arranged between both frames and frame Vi remains high wherever the target encoding frame is arranged between both frames. This is because the correlation between frame Vn and frame Vi is high. In such a case, the prediction error value and the generation code amount do not change much both in the forward prediction encoding in which frame Vi is the reference frame and in the bi-directional prediction encoding in which frame Vi and frame Vn are the reference frames. Therefore, in such a case, the forward prediction encoding, rather than the bi-directional prediction encoding, can suppress the generation of reference frame update processing in which frame Vn is the reference frame.
The determination described above reduces the calculation processing amount during the encoding processing while suppressing a decreased in encoding efficiency.
The following describes this determination method more in detail with reference to
First,
First, the frame V1 (608), the first frame, is encoded as an I-picture without referencing other frames. The frame V1 (608) is saved in the memory as reference image data. Next, the frame V2 (611), which is several frames apart from the frame V1 (608), is encoded as a P-picture with the frame V1 (608) as the reference frame. At this time, the correlation between the frame V2 (611) and the frame V1 (608) is also determined. In the case of
Next, the next P-picture V3 (614) is encoded using the new reference frame V2 (611). At this time, the correlation between the P-picture V3 (614) and the reference frame V2 (611) is determined. In the example shown in
Thus, when the correlation between the P-picture frame V3 (614) and the reference frame V2 (611) in
Next,
First, the frame V1 (608) is encoded as an I-picture without referencing other frames. After that, the frame V2 (611) is encoded as a P-picture by referencing the frame V1 (608). At this time, the correlation between the frame V1 (608) and the frame V2 (611) is determined in the same way as in the example in
Next, because the reference image is still the frame V1 (608), the frame V3 (614), which is a P-picture, is encoded by referencing the frame V1 (608). At this time, the correlation between the frame V1 (608), which is the current reference frame, and the frame V3 (614), which is the current target encoding image, is determined. In the example shown in
When an encoded stream is generated using an example of the encoding method shown in
At this time, if the multiple P-picture frames (for example, frame (609), frame (610), and frame (611) in
At this time, if the one or more B-picture frames are followed immediately by a P-picture frame (for example, frame (614) in
Next,
First, the frame V1 (608) is encoded as an I-picture without referencing other frames. After that, the frame V2 (611) is encoded as a P-picture by referencing the frame V1 (608). The correlation between the frame V1 (608) and the frame V2 (611) is high in the same way as in
Next, when the P-picture frame V3 (614) is encoded, the reference image is still the frame V1 (608) as it is in
As a result, in the example in
An example of the encoding processing method in this embodiment has been described for multiple cases, in which the predetermined frames of moving image data are configured in different correlations, with reference to
Therefore, with a frame, which is several frames after the reference frame, as a P-picture, the method described above determines the correlation between this P-picture and the reference frame to determine if the reference frame is to be updated and, thereby, reduces the number of times the update processing is performed.
Next, the following describes an example of a moving image encoding apparatus in this embodiment with reference to
A moving image encoding apparatus (300) in
That is, the moving image encoding apparatus (300) shown in
The picture structure storage memory (311) holds information on the current picture structure (for example, whether the structure is a PB structure or a PP structure). When this information is updated or the transmission of this information is requested, the information is sent to the predicted image generation unit (302) and the reference frame update determination processing unit (305). The predicted image generation unit (302) encodes an encoding image based on the notified picture structure information.
The reference frame update determination unit (305) has not only the function of the reference frame update determination unit (105) in
If it is necessary to hold the stream information, corresponding to the encoding frame, temporarily in the memory as a result of the change in the picture structure, the information is held in the encoding result primary storage memory (313). After that, when the part of the encoded stream that is held is combined with the output encoded stream for output, the part is output from the encoding result primary storage memory (313) and is combined with the current output encoded stream for output.
Next, the following describes an example of the structure of the reference frame update determination processing unit (305) with reference to
The update determination processing unit (403) in the second embodiment also determines the correlation between the target encoding image and the reference frame using the fixed determination criterion information, local determination criterion information, and statistical determination criterion information in the same way as in the first embodiment.
Note that, in the second embodiment, the update determination processing unit (403) performs the reference frame update determination processing only when the current target encoding frame is a frame V in
If the current target encoding frame is a frame V in
First, a frame in the description in
Thus, if the current target encoding image is a frame V in
If the determination result of the update determination processing unit (403) is that the correlation between the target encoding image and the reference image is high, the picture structure determination unit (408) issues the picture structure control signal, which changes the picture structure to the PP structure, to the picture structure change processing unit (312) in
It is also possible that the picture structure determination unit (408) is made independent of the reference frame update determination processing unit (305) and, in addition, the picture structure determination unit (408), picture structure change processing unit (312), and picture structure storage memory (311) are integrated into one prediction-direction determination unit. This is because, if a frame V is the current target encoding image, deciding the picture structure is equivalent to deciding the picture type of a frame Z that is temporally between the target encoding image and the reference frame. In other words, this is equivalent to deciding the prediction method, forward prediction or bi-directional prediction, of a predicted image used in the encoding processing of the frame Z.
Next,
First, one target encoding frame is encoded (801). This processing (801) is performed by the operation of the predicted image generation unit (302), subtracter (109), encoding processing unit (103), and variable-length encoding unit (104) included in the moving image encoding apparatus (300) in
On the other hand, if it is determined in the update determination step (802) that the encoding frame is the P-picture of a V frame, the update determination processing unit (305) uses the determination method in this embodiment described above to determine if the reference frame must be updated (805).
If it is determined by the update determination processing unit (305) in the determination step (805) that the reference frame must be updated, control is passed to the determination step (806). In the determination step (806), the picture structure determination unit (408) determines if the picture structure is the PP structure or the PB structure. If the picture structure is the PP structure, the picture structure is changed (807). This processing is performed as follows. That is, the picture structure determination unit (408) sends the picture structure control signal, which changes the picture structure from the PP structure to the PB structure, to the picture structure change processing unit (312). Next, the picture structure change processing unit (312) changes the current picture structure information, held in the picture structure storage memory (311), from the PP structure information to the PB structure information. The above steps complete the picture structure change processing (807). Next, control is passed to the reference frame update processing (803). If it is determined in the determination step (806) that the picture structure is the PB structure, the picture structure is not changed but control is passed to the reference frame update processing (803). That is, if the reference frame must be updated after the frame V is encoded as a P-picture (if the correlation between the reference frame and the encoding frame is low), the picture structure is set to the PB structure regardless of the current picture structure. After that, the reference frame is updated (803) and control is passed to the determination step (810). If it is determined in the determination step (810) that all frames are not yet encoded, control is passed back to the encoding processing (801). If all frames are encoded, the encoding processing is terminated.
If, in the determination step (805), it is determined by the update determination processing unit (305) that the reference frame need not be updated (805), control is passed to the determination step (808). In the determination step (808), the picture structure determination unit (408) determines whether the picture structure is the PP structure or the PB structure. If the picture structure is the PB structure, the picture structure is changed (809). This processing is performed as follows. That is, the picture structure determination unit (408) sends the picture structure control signal, which changes the picture structure from the PB structure to the PP structure, to the picture structure change processing unit (312). Next, the picture structure change processing unit (312) changes the current picture structure information, held in the picture structure storage memory (311), from the PB structure information to the PP structure information. The above steps complete the picture structure change processing (809). Next, control is passed to the determination step (810). If it is determined in the determination step (808) that the picture structure is the PP structure, the picture structure is not changed but control is passed to the determination processing (810). That is, if the reference frame need not be updated after the frame V is encoded as a P-picture (if the correlation between the reference frame and the encoding frame is high), the picture structure is set to the PP structure regardless of the current picture structure. If it is determined in the determination step (810) that all frames are not yet encoded, control is passed back to the encoding processing (801). If all frames are encoded, the encoding processing is terminated.
If the encoding frame is the P-picture of a frame Z or the B-picture of a frame Z in the determination step (802), control is passed to the determination step (810). If it is determined in the determination step (810) that all frames are not yet encoded, control is passed back to the encoding processing (801). If all frames are encoded, the encoding processing is terminated.
An encoded stream for which the encoding processing (801) is performed is output as necessary. In this case, depending upon the picture structure, a part of the encoded stream is held temporarily in the encoding result primary storage memory (313). By doing so, the encoded stream is output with its output time or output order adjusted.
The operation of the update determination processing unit and the moving image encoding apparatus in the second embodiment described above or the operation of the components in the flow of the encoding processing performed by them may be implemented by the autonomous operation of the components or by the instruction from the control unit (350). The control unit (350) and the software may work together to implement the operation.
The update determination processing unit and the moving image encoding apparatus in the second embodiment described above or the encoding processing performed by them controls whether or not a reference frame is to be updated also in the moving image encoding processing in which B-pictures are used. This control operation reduces the number of times the reference frame update processing is performed, for example, in a slow-moving scene and reduces the generation amount of calculation processing associated with the reference frame update processing such as the decoding processing, memory transfer processing, or other processing using decoded image data. In addition, controlling the number of reference frame update processing operations and changing a part of the picture structure of encoded data reduce the calculation processing generation amount and, at the same time, suppresses a decrease in encoding efficiency.
That is, the second embodiment of the present invention reduces the amount of calculation processing during the encoding processing while suppressing a decrease in encoding efficiency.
Next, an example of the encoding processing in the first embodiment and the second embodiment described above will be described with reference to
First, the following describes an example of the encoding processing in the first embodiment with reference to
In the description of
The following describes an example of reference frame update determination processing performed by the reference frame update determination processing unit (105) in the first embodiment using the local determination criterion information and the fixed determination criterion information. Assume that expression 2 is used for the local determination criterion in this figure. That is, the reference frame update processing is performed using the threshold calculated by multiplying the generation code amount of the frame following the initial reference frame by the constant σ. The threshold of the fixed determination criterion is the generation code amount SH(904). This generation code amount SH(904) corresponds to β in expression 1.
First, when the encoding is started, FB0×σ (903) becomes the local determination criterion threshold based on the generation code amount FB0(902). If the code amount of the target encoding frame is lower than the local determination criterion, the reference frame is not updated. In this figure, the code amount reaches the local determination criterion threshold FB0×σ (903) of the encoding frame at the frame number f1. At this time, the reference frame update determination processing unit (105) performs the determination processing and, as a result, the reference frame is updated. After the reference frame is updated, the correlation between the reference frame and the encoding frame becomes high. So, at frame f1+1 that is the next frame of the new reference frame, the generation code amount decreases to the level of frame f1. This means that, in the first embodiment, the determination based on the local determination criterion in a slow-moving scene reduces the number of times the reference frame update processing is performed and reduces the calculation processing amount during the encoding processing while suppressing a decrease in encoding efficiency.
Next, the data (901) suddenly moves fast at frame f2 and the generation code amount is increased. First, the following describes the processing that is performed assuming that the reference frame update determination processing unit (105) does not use the fixed determination criterion in this figure. In this case, because the generation code amount of frame f2 exceeds the local determination criterion threshold FB0×σ (903), the reference frame is updated. The frame f2 becomes the new reference frame. Therefore, the new local determination criterion threshold FB2×σ (906) is determined from the generation code amount FB2 (905) of the frame f2+1 that is the next frame of frame f2. In this case, the generation code amount data increases like the data (908) indicated by the dotted line starting at the point (907).
Next, the following describes a case in which the reference frame update determination processing unit (105) sets the threshold SH(904) as the fixed determination criterion in this figure. In this case, the generation code amount of frame f2 exceeds not only the local determination criterion threshold FB0×σ (903) but also the fixed determination criterion threshold SH(904), the reference frame is updated. Next, because the generation code amount of FB2(905) of frame f2+1 is smaller than the new local determination criterion threshold FB2×σ (906) but is larger than the fixed determination criterion threshold SH(904), the reference frame is updated in this case, too. After that, the reference frame is updated for each frame until the generation code amount of the encoding frame falls below the fixed determination criterion threshold SH(904). Therefore, when the fixed determination criterion is used, the generation code amount after the point (907) in this figure gets smaller than the data (908) described above, and the generation code amount changes like the data (909) indicated by the solid line. Therefore, in the first embodiment, the determination based on the fixed determination criterion in a fast-moving scene controls the number of times the reference frame update processing is performed, suppresses a decrease in encoding efficiency, and reduces the calculation processing amount during the encoding processing.
Next,
Referring to
In a slow-moving scene, the number of times the reference frame update processing is reduced in the same way as in
Next, an example of the encoding processing in the second embodiment will be described also with reference to
That is, also in the encoding processing in the second embodiment, the determination based on the local determination criterion in a slow-moving scene reduces the number of times the reference frame is updated and reduces the calculation processing amount during the encoding processing while suppressing a decrease in encoding efficiency.
Also in the encoding processing in the second embodiment, the determination based on the fixed determination criterion in a fast-moving scene controls the number of times the reference frame update processing is performed and suppresses a decrease in encoding efficiency to reduce the calculation processing amount during the encoding processing.
Also in the encoding processing in the second embodiment, the determination based on the statistical determination criterion suppresses a decrease in encoding efficiency and, at the same time, reduces the number of times the reference frame update processing is performed and reduces the calculation processing amount during the encoding processing even in a case in which an input image that is not pre-assumed is received.
As described above, the first embodiment and the second embodiment of the present invention can reduce the calculation processing amount during the encoding processing while suppressing a decrease in encoding efficiency.
In the embodiments of the present invention described above, only a part of the picture structure, indicated as the PB structure or PP structure, can be changed. At this time, however, if the reference frame update frequency is low, it is also possible to prolong the GOP(Group of Picture) period or to prolong the P-picture insertion period. In this case, the encoding is more efficient depending upon the input data.
The encoding processing in the above embodiments has been described as frame-basis encoding processing. However, all embodiments described above can be applied to the field-basis encoding of interlace image signals. That is, for the description of field-basis encoding, “frame” in the description of the embodiments should be replaced by “field”.
The encoding processing method or the encoding apparatus in the embodiments of the present invention described above can also provide a technology for encoding moving image data at a high speed. The encoding method or the encoding apparatus can also provide a moving image encoding technology for encoding moving image data at a high speed while maintaining the image quality.
The encoding processing method or the encoding apparatus described in the embodiments of the present invention described above is efficiently used also for a device or a system for encoding slow-moving videos such as a monitor image or a video conference.
Any of the embodiments described above can be combined as one embodiment of the present invention.
The encoding processing method or the encoding apparatus in the embodiments of the present invention described above can reduce the calculation processing amount during the encoding processing while suppressing a decrease in encoding efficiency.
It should be further understood by those skilled in the art that although the foregoing description has been made on embodiments of the invention, the invention is not limited thereto and various changes and modifications may be made without departing from the spirit of the invention and the scope of the appended claims.
Number | Date | Country | Kind |
---|---|---|---|
2006-306135 | Nov 2006 | JP | national |