The present application claims priority from Japanese application JP2006-326447 filed on Dec. 4, 2006, the content of which is hereby incorporated by reference into this application.
The present invention relates to a frame rate conversion technique for conversion of a video signal frame frequency (hereinafter, referred to “frame rate”).
For example, EP0909092A2 and the corresponding JP-A-11-112939 disclose a technique for converting a frame rate by searching a motion vector between a plurality of frames of an input video signal, a position of an image contained in the preceding and subsequent frame image according to the searched motion vector, generating a new frame image, and inserting the new frame image between the preceding and the subsequent frame.
The frame rate conversion technique using a motion vector disclosed in EP0909092A2 and JP-A-11-112939 searches a motion vector between a plurality of frames of an input video signal and generates a new block according to the motion vector so as to generate an interpolation frame.
Here, if the search range of the motion vector for searching the motion vector is to be reduced so as to reduce the calculation process amount in searching the motion vector, the ratio of the motion vectors exceeding the search range relatively increases in the motion vectors of the input video signal. This causes a motion vector search mistake and increases the frames of the interpolation image of a low image quality, which in turn lowers the quality of video.
It is therefor an object of the present invention to reduce the calculation process amount in the frame conversion process while suppressing lowering of the image quality.
In order to achieve the aforementioned object, an embodiment of the present invention includes: an input unit for inputting the video signal, a video interpolation unit for generating an interpolation frame and performing an interpolation process of the video signal, and a control unit for controlling the generation process of the interpolation frame by the video interpolation unit, wherein the video interpolation unit performs the interpolation frame generation process by using a plurality of methods, and the control unit controls switching between the plurality of methods.
According to the present invention, it is possible to reduce the calculation process amount in the frame rate conversion process while suppressing lowering of the image quality of a video.
Other objects, features and advantages of the invention will become apparent from the following description of the embodiments of the invention taken in conjunction with the accompanying drawings.
Description will now be directed to embodiments of the present invention with reference to the attached drawings.
Moreover, in the respective drawings, like components having like symbols have the same functions.
Moreover, the expressions “a frame” and “a predetermined region in a frame” in the Description and the Drawings may include meanings of “a frame image” and “an image in a predetermined region of the frame”, respectively.
Firstly, explanation will be given on a first embodiment of the present invention with reference to the drawings.
The frame rate conversion apparatus 100, for example, includes: an input unit 1 for inputting an input video signal; a video interpolation unit 2 for changing the frame rate of the input video signal by performing an interpolation frame generation process concerning the input video signal for the input video signal by using a plurality of methods; an output unit 3 for outputting the video signal for which the frame rate has been changed, as an output video signal; a motion vector search unit 4 for searching a motion vector for the input video signal to be inputted to the input unit 1; a histogram calculation unit 6 performing a histogram calculation process on the motion vector searched by the motion vector search unit 4; a storage unit 8 for storing the motion vector searched by the motion vector search unit 4, the histogram distribution information calculated by the histogram calculation unit, and judgment reference information; and a control unit 8 for switching the method of the interpolation frame generation process of the video interpolation unit 2 by performing a judgment process by using the histogram distribution information and the judgment reference information.
Hereinafter, detailed explanation will be given on each of the components of the frame rate conversion apparatus 100.
It should be noted that the operations of the respective components of the frame rate conversion apparatus 100 may be autonomous operations of the respective components as follows. Moreover, for example, the control unit 7 may be realized by cooperating with software stored in the storage unit 8.
Firstly, an input video signal is inputted to the input unit 1. The input unit 1 transmits an input video signal to the video interpolation unit 2. The video interpolation unit 2 performs an interpolation process on the input video signal acquired from the input unit 1. Next, the video interpolation unit 2 transmits the video signal converted to a desired frame rate by performing the interpolation process, to the output unit 3. The output unit 3 outputs the video signal converted to the desired frame rate from the frame rate conversion apparatus 10.
Here, it is assumed that the interpolation process performed by the video interpolation unit 2 selectively performs a plurality of types of generation method for its interpolation frame generation method. Detailed explanation will be given later on the interpolation frame generation method or the interpolation process method performed by the video interpolation unit 2.
Next, for example, when the video interpolation unit 2 performs an interpolation process by using a motion vector, the entire region or a part of the region of at least two video frames, i.e., temporally preceding and subsequent two frames is inputted from the input unit 1 to the motion vector search unit 4.
Here, the motion vector search process by the motion vector search unit 4 is performed as follows. For example, a motion vector between video frames is searched for each block formed by a plurality of pixels or each pixel. The motion vector search method may be a known one such as the block matching and is not limited to a particular method. Next, the motion vector search unit 4 transmits the motion vector obtained by the search process, for example, to the video interpolation unit 2, the histogram calculation unit 6, the storage unit 8, and the like.
Here, in the motion vector search process by the motion vector search unit 4, the size of the range for searching a motion vector in the motion vector search object frame affects the calculation process amount of the motion vector search process. Here, in order to reduce the calculation process amount of the motion vector search process, for example, a small range may be set for searching the motion vector. However, as the motion vector search range becomes smaller, the probability of generation of a motion vector search mistake is increased. It should be noted that details on the motion vector search range and the motion vector search mistake will be given later.
Next, when the video interpolation unit 2 performs an interpolation process by the interpolation process method using a motion vector, the video interpolation unit 2 uses a motion vector acquired from the motion vector search unit 4 to perform the interpolation process of the input video signal.
Moreover, the histogram calculation unit 6 calculates a histogram distribution by counting motion vectors for each motion direction and motion amount in each screen or each predetermined region by using the motion vectors acquired from the motion vector search unit 4. For example, the histogram distribution calculation process may be performed as follows. The motion vectors acquired from the motion vector search unit 4 are stored by the histogram calculation unit 6 itself so as to calculate the histogram distribution. Moreover, it is possible to calculate the histogram distribution by using the motion vectors acquired from the motion vector search unit 4 and stored in the storage unit 8 and to store the calculated histogram information in the storage unit. In this case, the histogram calculation unit 6 need not store the information.
Next, the control unit 7 acquires the histogram distribution information calculated by the histogram calculation unit 6 from the histogram calculation unit 6 or the storage unit 8. Moreover, the control unit 7 acquires judgment reference information from the storage unit 8. Here, the control unit 7 transmits a switching instruction signal to switch the type of interpolation frame in accordance with the judgment result, to the video interpolation unit 2.
For example, the judgment reference information includes a threshold value of the number of motion vectors satisfying a predetermined condition, a threshold value of a ratio of the histogram distribution of the number of the motion vectors satisfying the predetermined condition against a modulus, and the predetermined condition (such as a predetermined motion amount condition, a predetermined direction condition, or a combination of them). Moreover, the judgment reference may be a condition capable of detecting whether the probability that the motion vector search unit 4 has made a vector search mistake is high. In this case, in accordance with the probability of generation of the motion vector search mistake, it is possible to select the interpolation frame generation process in the video interpolation unit 2. For example, when the probability of generation of the motion vector search mistake is low, it is possible to select a high-quality interpolation process method using a motion vector. Moreover, for example, when the probability of generation of the motion vector search mistake is high, it is possible to select an interpolation process method not using a motion vector. In this case, it is possible to employ a control method to prevent lowering of the interpolation frame image quality by a motion vector search mistake. It should be noted that detailed explanation on the judgment reference information and an example of the judgment process by the control unit 7 will be given later.
Next, the video interpolation unit 2 which has acquired the switching instruction signal for the interpolation process from the control unit 7 changes the interpolation process method according to the switching instruction signal.
As has been described above, the frame rate conversion apparatus 100 according to the present embodiment selects a plurality of interpolation process methods and performs an interpolation process for an input video signal and can output an output video signal having a frame rate modified by the interpolation process.
Moreover, the frame rate conversion apparatus according to the present embodiment performs a switching judgment process of a plurality of interpolation process methods by using a motion vector acquired from an input video signal and a predetermined judgment reference so as to prevent lowering of the image quality of an interpolation frame generated in the interpolation process and prevent lowering of the image quality of the output video.
Next, referring to
As has been described above, the video interpolation unit 2 performs a plurality of types of interpolation method while switching from one to another according to a switching signal from the control unit 7, for example. The video interpolation unit 2 according to the first embodiment, for example, performs three types of the interpolation frame generation method shown in
It should be noted that the explanation on
Firstly, referring to
Here, in the method for generating the interpolation frame 22 according to the present method, for example, a pixel value of the pixel P1 on the interpolation frame 22 is defined by a pixel value of the pixel Pa1 on the preceding frame 21 at the same spatial pixel position as the pixel P1 within the frame. In the same way, a pixel in the interpolation frame 22 is made to be a pixel value of the pixel positioned at the corresponding position on the preceding frame 21, thereby generating the interpolation frame 22 as the same image as the preceding frame 21. This method is characterized in that no vector search or linear interpolation process is required and the calculation amount is minimized. Moreover, this method has an advantage that the image quality is not degraded as compared to the preceding frame. However, in this method, depending on the input image, image degrading of so-called motion judder may be caused. That is, an image which should be moving while displayed appears alternately in motion and as a still image.
It should be noted that in the method of
Next, referring to
In the method for generating the interpolation frame 22 according to the present method, for example, a pixel value of a pixel P2 on the interpolation frame 22 is calculated by linear interpolation by using a pixel Pa2 on the preceding frame 21, which pixel is positioned at the same spatial pixel position as the pixel P1 in the frame and a pixel Pb2 on the subsequent frame 23, which pixel is positioned at the same spatial pixel position as the pixel P1 in the frame.
More specifically, the Expression (1) given below can be used for the calculation.
Since this method does not require the motion vector search, the calculation amount can be minimized as compared to the method using the motion vector search. Moreover, as compared to the interpolation process method repeating the temporally preceding frame explained in
It should be noted that when generating a plurality of interpolation frames in the method of
Next, referring to
In the method for generating the interpolation frame 22 according to this method, for example, when calculating a pixel value of a pixel P3 on the interpolation frame 22, the video interpolation unit 2, for example, uses a motion vector acquired from the motion vector search unit 4 of
Here, In the interpolation process by the present method, the pixel value of the pixel P3 on the interpolation frame 22 is calculated, for example, by using the pixel value of the pixel Pa3 as the start point of the motion vector 24 and the pixel value of the pixel Pb3 as the end point of the motion vector 24. For example, the pixel value of the pixel Pa3 may be directly used as the pixel value of the pixel P3. Moreover, the pixel value of the pixel P3 may be calculated by replacing “P2” by “P3”, “Pa2” by “Pa3”, and “Pb2” by “Pb3” in Expression (1).
The method of
Next, explanation will be given on the case when the method of
Consequently, when such a motion vector search mistake has occurred, the image quality of the interpolation frame is degraded.
The video interpolation unit 2 according to the present embodiment can appropriately obtain both of the advantage to achieve a high image quality and the advantage to reduce the calculation amount by selectively switching between the interpolation methods of
Next, referring to
The judgment reference information 30 may be stored, for example, in the storage unit 8 in
It should be noted that the judgment reference information 30 shown in
For example, the data stored in row 32 has a data name “threshold value 1”, a data type “vector occupation ratio in region A”, and a content “30%”. The judgment process using the data will be detailed later.
Moreover, for example, the data stored in row 33 has a data name “limit period 1”, a data type “switching limit period”, and a content “1 sec”. The data is, for example, limit period data for limiting the switching process of the interpolation process by a predetermined period of time. The judgment process using the data will be detailed later.
Moreover, for example, the data stored in row 34 has a data name “judgment period 1”, a data type “judgment period”, and a content “0.3 sec”. The data is on a predetermined period used, for example, for judging the switching process of the interpolation process. The judgment process using the data will be detailed later.
Moreover, for example, the data stored in row 35 has a data name “threshold value 2”, a data type “vector occupation ratio in region A”, and a content “20%”. The data is on a threshold value like “threshold value 1”. The judgment process using the data will be detailed later.
Moreover, for example, the data stored in row 36 has a data name “acquisition signal 1”, a data type “acquisition signal condition”, and a content “signal B contained”. The data is, for example, a judgment reference condition of the switching process of the interpolation process and indicates that a switching process is executed if the input video signal contains the signal B. The judgment process using the data will be detailed later.
Moreover, for example, the data stored in row 37 has a data name “threshold value 2”, a data type “vector occupation ratio in region A”, and a content “15%”. The data is, for example, a combination of a judgment condition of a signal contained in the input video signal and a condition of threshold value data using the number of motion vectors. The judgment process using the data will be detailed later.
The control unit 7 according to the first embodiment of the present invention acquires necessary information from the judgment reference information 30 explained above, judges whether to switch the interpolation process method, and transmits a switching signal of the interpolation process method to the video interpolation unit 2. Thus, the video interpolation unit 2 according to the first embodiment of the present invention can realize switching of an appropriate interpolation process method according to various conditions.
Next, referring to
Moreover, a region 43 is a region having a an external predetermined width within the search range 42. The region 43 in
Next, explanation will be given on values indicated in each pixel in
Here,
The histogram distribution calculation process may be performed by the histogram calculation unit 6 by using, for example, the motion vectors searched by the motion vector search unit 4 and stored in the storage unit 8.
The histogram distribution shown in
Next,
Here, if the histogram distributions of
Accordingly, in order to eliminate the trade-off and simultaneously reduce the calculation amount and prevent lowering of an image quality of the interpolation frame, the frame rate conversion apparatus performs a process as follows. That is, for example, the motion vector search unit 4 performs a motion vector search process within a narrower search range than in the conventional method like the search range 40. Here, the video interpolation unit 2 generates an interpolation frame of a higher image quality by using the interpolation method using the motion vector shown in
When the aforementioned interpolation process is performed, for example, by using a narrower range requiring a smaller calculation amount than in the conventional method, it is possible to prevent the affect from lowering of the image quality of the interpolation frame caused by a motion vector search mistake even if a video having a plenty of motions is inputted.
Lowering of the image quality caused when the method of
Referring to
Here, the control unit 7 compares the judgment parameter to the threshold value contained in the judgment reference information 30 stored in the storage unit 8 so as to perform a judgment process. That is, the data “threshold value 1” contained in row 32 of the judgment reference information 30 shown in
Accordingly, the control unit 8 compares the aforementioned judgment parameter to the threshold value. That is, if the judgment parameter is lower than 30%, the video interpolation unit 2 performs control so as to execute the interpolation process method shown in
Next, explanation will be given on the advantages obtained by the switching process according to the present embodiment. The histogram distributions shown in
As shown in
Here, in the example of
Accordingly, the calculation process within the search range 41 and outside the search range 42 has a low efficiency. In particular, when the switching control of the interpolation process method of the present embodiment is not performed, the search range should be set to a greater range with a margin so as to prevent occurrence of a motion vector search mistake. Consequently, the calculation process efficiency is further lowered.
Here, the motion vector search unit 4 of the present embodiment narrows the motion vector search range, for example, to the search range 42 so as to reduce the calculation amount. In addition to this, the control unit 7 performs the aforementioned interpolation process method switching process for the interpolation frame having a plenty of vector search mistakes corresponding to a portion outside the search range 42. Thus, by switching to the interpolation process method not using a motion vector, it is possible to prevent lowering of the frame image quality.
Accordingly, in the examples of the aforementioned
Moreover, the motion vector search range may be set by using a histogram statistically calculated like in
Moreover, the region 43 shown in
Moreover, the respective search ranges, the regions, the motion vector histogram distributions shown in
Moreover, the motion vector histogram distributions shown in
Moreover, the judgment parameter in
By using the interpolation process switching judgment process using the motion vector histogram distribution explained above, it is possible to reduce the motion vector search range and the calculation process amount. Simultaneously with this, it is possible to prevent lowering of the image quality caused by a motion vector search mistake.
Accordingly, in the first embodiment of the present invention, it is possible to reduce the calculation process amount in the frame rate conversion process while suppressing lowering of the video image quality.
Next, explanation will be given on a second embodiment of the present invention with reference to the drawings. The frame rate conversion apparatus according to the second embodiment is identical to the frame rate conversion apparatus according to the first embodiment except for that a predetermined period is set for the switching limit period concerning the switching judgment process or the judgment process performed by the control unit 7. Alternatively, in the frame rate conversion apparatus of
Referring to
Here, in
Moreover, the interpolation process method transitions 62, 63 are obtained by control of the switching process performed by the control unit 7 on the video interpolation unit.
The interpolation process method transition 62 shows a case when the control unit does not set a predetermined limit period for the switching. The switching process is, in principle, performed by comparing the judgment parameter to the threshold value 1 depicted. The threshold value 1 is identical to “the threshold 1” in the first embodiment. For example, if the judgment parameter value is smaller than 1, the control unit 7, in principle, selects the interpolation process method using a motion vector as the interpolation process method. Moreover, for example, if the judgment parameter value is greater than 1, the control unit 7, in principle, selects the interpolation process method not using a motion vector as the interpolation process method.
The interpolation process method transition 63 indicates a case when the control unit sets a predetermined limit period for the switching. In this case, in addition to the switching based on the principle, for example, the control unit may set a predetermined limit period for the switching as an exception. Alternatively, the control unit may set a predetermined judgment period for the switching judgment as an exception.
Hereinafter, explanation will be given on the case when the control unit 7 sets a predetermined limit period for the switching and the case when the control unit 7 does not set it.
Firstly, explanation will be given in time series on the case when the control unit does not set a predetermined limit period for the switching, by using the judgment parameter transition 61 and the interpolation process method transition 62 of
Here, in the interpolation process method transition 62 in
In order to improve this phenomenon, the control unit 7 sets a predetermined limit period for the switching. This will be explained in time series by using the judgment parameter transition 61 and the interpolation process method transition 63 in
As has been described above, by setting a switching limit period in the switching judgment process of the interpolation process method, it is possible to limit the switching of the interpolation process method in a predetermined period. Thus, for example, even when the judgment parameter value changes to cross the level of the threshold value 1 by plurality of times within a short period of time, it is possible to prevent frequent occurrence of the switching process of the interpolation process method in a short period of time. Thus, it is possible to obtain an output video after the frame rate conversion process as a video of a natural motion.
Next, explanation will be given on the case using the judgment period 65 instead of the switching limit period. The aforementioned switching limit period 64 is a period during which no next interpolation switching process is performed after an interpolation process method switching process has occurred. On the other hand, during the judgment period 65, no interpolation process method switching process is performed at the time satisfying the judgment reference stored in the judgment reference information 30. The judgment period 65 is a period for performing the interpolation process method switching process when the state that the judgment reference is satisfied has continued for a predetermined period of time after a certain moment.
By using the judgment parameter transition 61 and the interpolation process method transition 63 in
Firstly, explanation will be given on the process from time t4. At time t4, the judgment parameter value becomes smaller than the threshold value 1. However, at this moment, the control unit 7 does not transmit the interpolation process switching signal. Next, at time t4 when the judgment period 65 has elapsed, the control unit 7 makes a judgment as follows. That is, judgment is made whether the judgment parameter value has satisfied the judgment reference during the judgment period 65. In other words, judgment is made whether the judgment parameter value was smaller than the threshold value 1 during the time from time t4 to time t5. In
Next, explanation will be given on the process performed from time t2. Here, it is assumed that the period from time t2 to t3 is shorter than the judgment period 65. Firstly, at time t2, the judgment parameter value becomes smaller than the threshold value 1. At this moment, the control unit 7 does not transmit the interpolation process switching signal. Next at time t3, the judgment parameter becomes greater than 1. Here, the interpolation process method has been set to the interpolation process method not using a motion vector and the control unit 7 does not transmit the interpolation process switching signal. Here, the period from time t2 to t3 is shorter than the judgment period 65. Accordingly, at time t3 before the judgment period 65 has elapsed after the time t2, the state that the judgment parameter value satisfies the judgment reference is terminated. For this, no interpolation process method switching process has been performed.
Accordingly, even when the judgment period 65 is used, like in the case using the switching limit period, for example, when the judgment parameter value crosses the level of the threshold value 1 by a plurality of times during the period from time t2 time t3, it is possible to prevent frequent occurrence of the interpolation process method switching process. Thus, it is possible to obtain an output video of a natural motion after the frame rate conversion process.
It should be noted that like the switching limit period 54, the judgment period may be acquired, for example, from the judgment reference information 30 stored in the storage unit 8 by the control unit 7. For example, it is possible to use the value of “0.3 sec” of “the judgment period 1” in row 34 of the judgment reference information 30 in
Moreover, a method other than the use of the aforementioned switching limit period 64 or the judgment period 65 may be used to obtain the same advantage. That is, it is possible to modify the threshold value used for the switching judgment process between a case when the judgment parameter value is increasing and a case when the judgment parameter value is decreasing.
That is, as shown in
Here, if the judgment parameter value is increasing, the control unit 7 uses the threshold value 1 for the switching judgment process. Moreover, for example, if the judgment parameter value is decreasing, the control unit 7 uses the threshold value 2 for the switching judgment process. That is, in the switching judgment process by the control unit 7, a hysteresis is provided for the judgment parameter value.
Explanation will be given on a case using a plurality of threshold values by using the judgment parameter transition 61 and the interpolation process method transition 63 in
Firstly, during the period from t0 to time t1, the judgment parameter is increasing. Accordingly, the control unit does not perform the switching process. Next, at time t1 when the judgment parameter value reaches the threshold value 1, the control unit 7 selects the interpolation process method not using a motion vector as the interpolation process method and transmits the interpolation process switching signal to the video interpolation unit 2. Moreover, at time t2, the judgment parameter is decreasing. That is, the judgement parameter value has reached the threshold value 1 but not reached the threshold value 2 and accordingly, the control unit 7 does not perform the switching process. Moreover, at time t2, the judgment parameter value is increasing. Accordingly, the control unit 7 uses the threshold value 1. However, the interpolation process method has been already set to the interpolation process method not using a motion vector and the control unit 7 does not perform the switching process. Next, at time t4 and after, the judgment parameter value is decreasing and the control unit 7 uses the threshold value 2. Accordingly, the control unit 7 does not perform the switching process at time t4. Next, at time t5, the control unit 7 selects the interpolation process method using a motion vector as the interpolation process method and transmits the interpolation process switching signal to the video interpolation unit 2.
As has been described above, when a plurality of threshold values are used, it is possible to obtain an advantage similar to that when using the switching limit period 64 or the judgment period 65. That is, for example, even when the judgment parameter value crosses the level of threshold value 1 by a plurality of times within a short period of time like a period from time t2 to time t3, it is possible to prevent frequent occurrence of the switching process of the interpolation process method in a short period of time. Thus, it is possible to obtain the output video after the frame rate conversion as a video of a natural motion.
By using any one of the aforementioned interpolation process switching judgment processes, it is possible to prevent frequent occurrence of the interpolation process method switching process in a short period of time and to obtain the output video after the frame rate conversion as a video of a natural motion.
Thus, according to the second embodiment of the present invention, it is possible to reduce the calculation process amount in the frame rate conversion process while suppressing lowering of the image quality of the video and to prevent the video after the frame rate conversion from becoming a video of an unnatural motion.
Next, explanation will be given on a third embodiment of the present invention with reference to the drawings. A frame rate conversion apparatus according to the third embodiment is identical to the frame rate conversion apparatus of the first or the second embodiment except for that other judgment reference information is used in the switching judgment process by the control unit 7.
For example, the control unit 7 in the third embodiment uses a judgment condition of row 30 stored in the judgment reference information 30 shown in
The control unit 7 uses the data stored in row 36 as the judgment reference. In this case, for example, the control unit 7 judges whether the input signal inputted to the input unit 1 of
By using the aforementioned judgment reference, for example, it is possible to detect a signal indicating a scene change contained in the input video signal and modify the interpolation process method. For example, this can be applied to a case when it is inappropriate to use the interpolation process using a motion vector such as at a scene change.
As for the signal used as the judgment reference in the acquired signal condition, for example, the input video signal may include a particular signal for performing the judgment process of the present embodiment. Alternatively, it is possible to use a signal contained in the existing input video signal as the judgment reference.
Moreover, for example, it is also possible to use as the judgment reference, a combination of the judgment reference condition of the signal contained in the input video signal like data stored in row 37 in the judgment reference information 30 in
Moreover, for example, the control unit 7 may change the respective threshold values stored in the judgment reference information 30 in accordance with the signal contained in the input video signal. In this case, there is no need of preparing a plurality of threshold values in the judgment reference information 30 in advance. Moreover, it is possible to set a threshold value having a further detailed correspondence with the signal contained in the input video signal as the judgment reference. For example, the threshold value may be a function having the numeric data contained in a signal contained in the input video signal as a variable.
Thus, according to the aforementioned third embodiment, it is possible to reduce the calculation process amount in the frame rate conversion process while suppressing lowering of an image quality of the video and to prevent execution of an inappropriate interpolation process for the frame rate conversion process.
Next, explanation will be given on a fourth embodiment of the present invention with reference to the attached drawings. A frame rate conversion apparatus according to the fourth embodiment is identical to the frame rate conversion apparatus of the first to the third embodiment except for that an independent interpolation process method is executed in a plurality of regions in the frame.
The fourth embodiment is used, for example, in a case when different videos are to be displayed in two video regions C71 and D72 in one video frame 70. The control unit 7 of the frame rate conversion apparatus of the fourth embodiment performs a judgment process using judgment reference information in the video region C71 and the video region D72. Moreover, according to the judgment process result, an interpolation process method switching method for each of the regions is transmitted to the video interpolation unit 2. Here, the video interpolation unit 2 performs switching between the interpolation process methods for each of the regions.
Here, the judgment reference information may be shared by the switching judgment process of the interpolation process method in the video region C71 and the switching judgment process of the interpolation process method in the video region D72. Moreover, the judgment reference information corresponding to the respective regions may be generated by the control unit 7 and stored in the storage unit 8 so that they are used in the switching judgment process of each of the regions.
In this embodiment, an independent interpolation process is executed in each of the two video regions C71 and D72 in the video frame 70. In this case, for example, when displaying a video having comparatively little motion in the video region C71 and a video having an intensive motion exceeding a motion vector search region in the video region D72, it is possible to use the interpolation method using a motion vector in the video region C71 and the interpolation process method based on the linear interpolation not using a motion vector in the video region D72. Thus, it is possible to perform the frame rate conversion by using different interpolation processes in the respective regions.
Explanation has been given on the case of two regions in a video frame. However, the present embodiment is not limited to two regions but can be applied to a case of more regions.
The aforementioned information on the region positions of a plurality of different regions having different image motions in the same frame may be set in advance and stored in the storage unit 8. Moreover, the motion vector calculated by the motion vector search unit 4 and stored in the storage unit 8 may be statistically evaluated by the control unit 7 so as to decide the range of each region.
According to the fourth embodiment thus far explained, even for the input video including a plurality of regions having different image motions in the same frame, it is possible to reduce the calculation process amount in the frame rate conversion process while suppressing lowering of the image quality of the video.
Next, explanation will be given on a fifth embodiment of the present invention. A display apparatus according to the fifth embodiment includes the frame rate conversion apparatus according to the first to the fourth embodiment as a frame conversion unit.
It should be noted that operations of the respective components of the display apparatus 800 explained below may be autonomous operations of the respective components or may be controlled by the control unit 810 connected to the respective components. This can be realized by cooperation of the control unit 810 with software.
Here, the antenna 81 receives a radio wave from outside and sends the received radio wave to the tuner unit 82. The tuner unit 82 converts the radio wave received from the antenna 81 into a video signal and sends the converted video signal to the signal processing unit 83. The signal processing unit 83 subjects the video signal acquired from the tuner unit 82 to a signal process and sends the signal-processed video signal to the display unit 84. The display unit 84 displays the video signal acquired from the signal processing unit 83.
Here, the radio wave received by the antenna 81 may be, for example, a TV broadcast wave.
Here, the control unit 810 sends a control signal to the signal processing unit 83 or the frame rate conversion unit 85 so as to control the signal processing unit 83 and the frame rate conversion unit 85 to perform the frame rate conversion process on the video signal received from the tuner unit 82.
Here, the signal processing unit 83 which has acquired the control signal from the control unit 810 sends the video signal received from the tuner unit or the video signal subjected to signal processing by the signal processing unit 83 to the frame rate conversion unit 85. Next, the frame rate conversion unit 85 which has acquired the video signal from the signal processing unit 83 executes the frame rate conversion process identical to the one performed by the frame rate conversion apparatus according to the first to the fourth embodiment or a frame rate conversion process based on a combination of the processes. Next, the frame rate conversion unit 85 sends the video signal subjected to the frame rate conversion process to the signal processing unit 83. Next, the video signal acquired by the signal processing unit 83 from the frame rate conversion unit 85 or the video signal obtained by subjecting the video signal to a signal process is outputted to the display unit 84. The display unit 84 displays the video signal acquired from the signal processing unit 83.
Here, in the frame rate conversion process performed by the frame rate conversion unit 85, the same interpolation process switching judgment process as in the first to the fourth embodiment is performed. Here, the judgment reference information used in the switching judgment process may be stored in advance in the storage unit held by the frame rate conversion unit 85. Moreover, the control unit 810 may control the judgment reference information by setting and modifying it.
Moreover, in
Moreover, in the above-given explanation, the display apparatus 800 in
Moreover, when the input unit 86 is connected to a network such as the Internet via the aforementioned terminals, for example, it is possible to acquire a video signal such as the Internet broadcast.
Moreover, the display apparatus 800 may be, for example, a plasma television, a liquid crystal television, or a CRT television, or a projector, or an apparatus using other apparatus. Similarly, the display unit 84 may be, for example, a plasma panel module, an LCD module, or a projector apparatus.
Accordingly, the display apparatus 800 according to the present embodiment can realize a display apparatus for performing the frame conversion process of the first to the fourth embodiment on the acquired video signal and displaying it.
Consequently, in the fifth embodiment of the present invention, it is possible to realize a display apparatus capable of reducing the calculation process amount in the frame conversion process while suppressing lowering of the image quality of the video.
Next, explanation will be given on a sixth embodiment of the present invention with reference to the attached drawings. A display apparatus according to the sixth embodiment is a recording apparatus including the frame rate conversion apparatus according to the first to the fourth embodiment as a frame conversion unit.
It should be noted that the operations of the respective components of the recording apparatus 900 described below may be autonomous operations of the respective components or may be controlled by the control unit 810 connected to the respective components, or may be realized in cooperation between the control unit 810 and software.
The output unit 91 outputs a video signal acquired from the signal processing unit 83, for example, to other display apparatus connected to the output unit 91. Here, the output unit 91, for example, may include a terminal for digital-outputting a video signal or may include a terminal for analog-outputting such as a composite terminal and a component terminal. Alternatively, the output unit 91 may include a LAN connector for connection to the network or a USB connector. Furthermore, the output unit 91 may be a transmission unit for transferring data by a wireless way.
The video storage unit 92 records a video signal acquired from the signal processing unit 83. Moreover, the recorded video signal is outputted to the signal processing unit 83. Here, the video storage unit 92 may be, for example, a hard disc drive or a flash memory, or a removable media disc drive. Moreover, the video storage unit 92 may be an apparatus for recording a signal on a mobile medium.
Here, the signal processing unit 83 performs video I/O operation to/from the video storage unit 92 in addition to the functions and operations explained in the fifth embodiment. Here, the video signal inputted to the video storage unit 92 by the signal processing unit 83, for example, may be a video signal which has been subjected to the frame conversion process in the frame conversion unit 85. Moreover, the video signal may be a signal not subjected to the frame rate conversion process. Moreover, the signal processing unit 83 may output the video signal acquired from the video storage unit 92 after subjecting it to the frame rate conversion process in the frame rate conversion unit 85. Moreover, it is possible to output a video signal not subjected to the frame rate conversion process to the output unit 91.
Accordingly, in the recording apparatus 900, the frame conversion unit 85 can perform an appropriate frame rate conversion process on the video signal acquired from the antenna 81 or the input unit 86 and the video storage unit 92 can store the video signal after the frame rate conversion process.
Moreover, in the recording apparatus 900, the frame rate conversion unit 85 can perform an appropriate frame rate conversion process on the video signal to be stored in the video storage unit 92 and the video signal after the frame rate conversion process can be outputted from the output unit 91.
Moreover, the recording apparatus 900, for example, may be an HDD recorder, a DVD recorder, or a recording apparatus using other storage apparatus.
Accordingly, in the recording apparatus 900 according to the present embodiment, it is possible to realize a recording apparatus for performing the frame rate conversion process according to the first to the fourth embodiment on the acquired video signal and recording the signal.
Consequently, according to the fifth embodiment of the present invention, it is possible to realize a recording apparatus capable of reducing the calculation process amount in the frame rate conversion process while suppressing lowering of the video image quality.
Referring to
Moreover, it is assumed that the input video signal includes a preceding frame and a subsequent frame. Moreover, it is assumed that the input video signal after being subjected to the frame rate conversion process includes an interpolation frame in addition to the preceding frame and the subsequent frame.
Moreover, in
Here, the frame 1001 of the preceding frame and the image 1004 of the preceding frame are images where pixels of the pixel value W are arranged in the shape of character “A”. The center position of the character “A” in the horizontal direction is assumed to be the position E.
Next, the subsequent frame image 1003 and the subsequent frame image 1006 are both images where the pixels of the pixel value W are arranged in the shape of the character “A” but the center position of the image of the character “A” in the horizontal direction is changed to the position F.
Accordingly, in
By using the image 1002 and the image 1005, explanation will be given on the interpolation frame image obtained when the aforementioned input video signal is inputted to the frame rate conversion apparatus, the display apparatus, or the recording apparatus according to the respective embodiments of the present invention.
Firstly, the image 1002 is an image when an interpolation frame is generated by the interpolation process method using a motion vector. For example, the interpolation frame is generated by the method shown in
The ratio of the vector amount G against the vector amount H may be generated, for example, so as to satisfy the following expression (2) in the case of
However, the vector amount H only should be smaller than the vector amount G and may not satisfy the Expression (2).
Next, the image 1005 is an image when the interpolation frame is generated by the interpolation process method not using a motion vector. For example, the interpolation frame image is generated by the video interpolation unit 2 in
Here, the image 1005 shows a plurality of images each formed by pixels arranged in the shape of the character “A”. In this example, an image formed by pixels of pixel value M arranged in the shape of “A” is present at the position E and another image formed by pixels of pixel value N arranged in the shape of “A” is present at the position F. Here, in the case of the linear interpolation shown in
However, the pixel value M and the pixel value N should not be greater than the pixel value W and need not satisfy the Expression (3) and the Expression (4).
Moreover, the image 1005 shows a case that images arranged in the shape of the character “A” are not overlapped. Here, if the images are overlapped, the pixel value of the pixels of the overlapped portion may be, for example M+N. Moreover, the pixel value of the pixels at the overlapped portion may be one of M and N which is greater than the other.
As has been described above, by inputting an input video signal including the image of the preceding frame and the subsequent frame shown in
Next,
Moreover, below the graph of the transition 1101 of the vector amount G, three different types of switching control of the interpolation process method are shown: a transition 1102 of the interpolation process method, a transition 1103 of the interpolation process method, and a transition 1104 of the interpolation process method. Hereinafter, explanation will be given on the relationship between the respective interpolation process methods and the transition 1101 of the vector amount G.
Firstly, the interpolation process method transition 1102 shows a case when no switching control of the interpolation process method is performed. In this case, like the transition 1101, even when the vector amount G of the vector in the input signal is increased to the vector amount G3 like arrow 1105 from time t1, no switching of the interpolation process method occurs. Here, in order to check that no switching of the interpolation process methods occurs, for example, the vector amount G3 should be increased as much as possible.
Next, explanation will be given on the interpolation process method transition 1103 showing that the interpolation process method switching control is performed. Here, in case of
In the case when the interpolation process method is switched like the aforementioned interpolation process method transition 1103, the interpolation process method is switched by using a threshold value concerning the vector amount of the motion vector. For example, the same check can be performed in the frame rate conversion apparatus using the first embodiment of the present invention.
Next, the transition 1104 of the interpolation process method is also an example when switching of the interpolation process method is performed. Here, in this graph, the interpolation process transition from time t0 to time t3 is identical to the interpolation process method transition 1103. That is, at time t1, the interpolation process method using a motion vector is switched to the interpolation process not using a motion vector. Next, from time t3, the vector amount G is decreased as shown by arrow 1106. Here, unlike the interpolation process method transition 1103, in the interpolation process method transition 1104, the interpolation process method switching control occurs at time t5 when the vector amount G has become G1. For example, this interpolation process method transition is obtained when the interpolation process method switching control according to the second embodiment of the present invention is performed. The interpolation process method switching control according to the second embodiment of the present invention, as has been described above, includes a method using the switching limit period, the method using the judgment period, a method using a plurality of threshold values (method having a hysteresis), and the like. These judgment methods will be explained below.
In the transition 1101 of the vector amount G in the input video signal, the vector amount G is decreased from time t3. Assume that the decrease is started at time t3+T (T is a predetermined time) and all the times hereafter are delayed by T. That is, the transition of the vector amount G is G2 at the time t4+T, G1 at time t5+T, and 0 at time t6+T. Assume that the interpolation process method switching control occurs at time t5 like in the original transition 1101 even if an input video signal having the new transition of the vector amount G. In this case, the interpolation process method switching control is performed, for example, by using the switching limit period among the interpolation process method switching controls according to the second embodiment of the present invention. That is, the interpolation process method switching control occurs at the same time as in the original case regardless of that at all the times after time t3, the relationships between the vector amount G and the respective times are different from the original transition 1101. For example, when a switching control period having a length t4-t1 is used and the threshold value is G2, at time t1, the interpolation process method is switched from the interpolation process method using a motion vector to the interpolation process method not using a motion vector at time t1. Next, at time t4 when the period of t4-t1 has elapsed, if the vector amount G is not greater than G2, the interpolation process method is switched from the interpolation method using a motion vector to the interpolation process method not using a motion vector at time t4.
Next, explanation will be given on a similar case of the transition 1101 of the vector amount G in the input video signal where the time to start decrease of the vector amount G from G3 is delayed to t3+T (T is a predetermined amount) and all the times hereafter are delayed by T but the interpolation process method switching control occurs at time t5+T when the vector amount G has become G1.
In this case, for example, the method using the judgment period or the method using a plurality of threshold values (method having a hysteresis) is performed among the interpolation process method switching controls according to the second embodiment of the present invention.
Firstly, in the transition 1101 of the vector amount G in the input video signal, the value of the vector amount G is fixed as shown in the transition 1107 of the vector amount G immediately before time 5. Here, the value of the vector amount G is slightly greater than G1. Here, if no interpolation process method switching control occurs at time t5 and after time t5, it can be said that the interpolation process method switching control is the method using a threshold value (method having a hysteresis) according to the second embodiment. That is, in this case, when the vector amount T increases, at the threshold value G2, the interpolation process method is switched from the interpolation process method using a motion vector to the interpolation process method not using a motion vector. Next, when the vector amount G decreases, at the moment when the vector amount G has become the threshold value G1, the interpolation process method is switched from the interpolation process method not using a motion vector to the interpolation method using a motion vector.
Next, even when a switch is made to the transition 1107 of the aforementioned vector amount G, if an interpolation process method switching control occurs at time t5, the interpolation process method switching control is performed by the method using the judgment period according to the second embodiment. That is, in the case of interpolation process method switching control using G2 for the switching control threshold value and t5-t4 as the judgment period, if the vector amount G is not greater than the threshold value G2 before the moment t5 when t5-t4 has elapsed from the moment t4 when the vector amount G has decreased to the threshold value G2, the interpolation process method is switched from the interpolation process method not using a motion vector to the interpolation process method using a motion vector.
As has been described above, when the interpolation process method is switched like the transition 1104 of the interpolation process method, the interpolation process method is switched by using the threshold value concerning the vector amount of the motion vector. Moreover, the aforementioned method using the switching limit period, the method using the judgment period, or the method using a plurality of threshold values (method having a hysteresis) is performed. For example, the similar check can be performed in the frame rate conversion apparatus using the second embodiment of the present invention.
Thus, by using the check method explained with reference to
According to the frame rate conversion apparatus, the display apparatus, or the recording apparatus according to the respective embodiments of the present invention thus far described, for example, it is possible to provide a technique for performing a frame rate conversion of a video signal at a high speed. Moreover, it is possible to provide a frame rate conversion technique for encoding dynamic image data at a high speed while preventing lowering of the image quality of the video.
It should be noted that even a combination of any of the aforementioned embodiments may be an embodiment of the present invention.
According to the frame rate conversion apparatus, the display apparatus, or the recording apparatus of the aforementioned respective embodiments, it is possible to reduce the calculation process amount in the frame rate conversion process while suppressing lowering of the image quality of the video.
It should be further understood by those skilled in the art that although the foregoing description has been made on embodiments of the invention, the invention is not limited thereto and various changes and modifications may be made without departing from the spirit of the invention and the scope of the appended claims.
Number | Date | Country | Kind |
---|---|---|---|
2006-326447 | Dec 2006 | JP | national |