The present invention is related to frame interpolation, and more particularly, to a method for performing frame interpolation based on a single-directional motion and an associated non-transitory machine-readable medium.
With advancement of technology, requirements for frames per second (FPS) of an image frame sequence are getting higher and higher. For example, when a user is watching a video corresponding to an image frame sequence, if the FPS of the frame sequence is too low, the video displayed on a screen may shake or become blurred. As a result, a frame interpolation technology is utilized to make the video smoother and therefore improve user's viewing experience. For a case where the frame interpolation is performed between an image frame I0 and an image frame I1 that is different from the image frame I0 in an image frame sequence, a conventional frame interpolation method must obtain a bi-directional motion (i.e. both of a motion from the image frame I0 to the image frame I1 and another motion from the image frame I1 to the image frame I0) to perform a backward warping operation upon the image frame I0 and the image frame I1, respectively, for generating multiple backward-warped results. Afterwards, the multiple backward-warped results are blended into a target interpolated frame for the frame interpolation performed between the image frame I0 and the image frame I1. However, when only a single-directional motion (e.g. only one of the motion from the image frame I0 to the image frame I1 and the motion from the image frame I1 to the image frame I0) can be obtained due to some reasons (e.g. tradeoff between cost, computation, and/or power saving), some problems may occur (e.g. the frame interpolation operation cannot be successfully performed). As a result, a novel method for performing frame interpolation based on a single-directional motion is urgently needed.
It is therefore one of the objectives of the present invention to provide a method for performing frame interpolation based on a single-directional motion and an associated non-transitory machine-readable medium, to address the above-mentioned issues.
According to an embodiment of the present invention, a frame interpolation method for generating a third image frame interpolated between a first image frame and a second image frame is provided. The frame interpolation method comprises: performing motion estimation upon a first input image frame and a second input image frame, to obtain a single-directional motion, wherein the first input image frame is derived from the first image frame, and the second input image frame is derived from the second image frame; scaling the single-directional motion according to a time point of the third image frame, to generate a scaled motion; deriving a forward-warped result from a result of performing a forward warping operation and a first inverse operation upon the scaled motion; performing a second inverse operation upon the forward-warped result, to generate an inversed result; and generating the third image frame according to the first image frame, the second image frame, the forward-warped result, and the inversed result.
According to an embodiment of the present invention, a non-transitory machine-readable medium for storing a program code is provided. When loaded and executed by a processor, the program code instructs the processor to execute a frame interpolation module. The frame interpolation module is arranged to generate a third image frame interpolated between a first image frame and a second image frame, and includes a motion estimating module, a scaling module, a first processing module, an inverse module, and a second processing module. The motion estimation module is arranged to perform motion estimation upon a first input image frame and a second input image frame, to obtain a single-directional motion, wherein the first input image frame is derived from the first image frame, and the second input image frame is derived from the second image frame. The scaling module is arranged to scale the single-directional motion according to a time point of the third image frame, to generate a scaled motion. The first processing module is arranged to derive a forward-warped result from a result of performing a forward warping operation and a first inverse operation upon the scaled motion. The inverse module is arranged to perform a second inverse operation upon the forward-warped result, to generate an inversed result. The second processing module is arranged to generate the third image frame according to the first image frame, the second image frame, the forward-warped result, and the inversed result.
One of the benefits of the present invention is that, by the frame interpolation method of the present invention, under a condition that only a single-directional motion can be obtained due to some reasons (e.g. tradeoff between cost, computation, and/or power saving), frame interpolation can be performed successfully.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the following description and claims, which refer to particular components. As one skilled in the art will appreciate, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not in function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”.
It is assumed that an image frame sequence {I0, I1, I2, . . . , In} including image frames I0-In with a low frames per second (FPS) is input into the frame interpolation module 20, wherein “n” is a positive integer greater than 1 (i.e. n>1). The frame interpolation module 20 is arranged to perform the frame interpolation upon the image frame sequence {I0, I1, I2, . . . , In} to increase its FPS. In this embodiment, for better comprehension, a target image frame It that is interpolated between the image frames I0 and I1 in the image frame sequence {I0, I1, I2, . . . , In} by the frame interpolation method of the present invention is taken as an example.
In the motion estimation portion, the resizing module 200 may be arranged to resize the image frames I0 and I1, respectively, to generate resized image frames RI0 and RI1, for tradeoff between interpolation quality and any of cost, computation, and/or power. The motion estimation module 202 may be arranged to perform motion estimation upon a first input image frame F_IIF and a second input image frame S_IIF, to obtain a single-directional motion F0→1, wherein the first input image frame F_IIF is derived from the image frame I0, and the second input image frame S_IIF is derived from the second image frame I1, and the single-directional motion F0→1 indicates a motion from the image frame I0 to the image frame I1. In this embodiment, the resized image frames RI0 and RI1 act as the first input image frame F_IIF and the second input image frame S_IIF, respectively, but the present invention is not limited thereto. In some embodiments, at the expense of the interpolation quality, the image frames I0 and I1 may directly act as the first input image frame F_IIF and the second input image frame S_IIF, respectively.
In the motion compensation portion, the scaling module 204 may be arranged to scale the single-directional motion F0→1 according to a time point of the target image frame It to generate a scaled motion F0→1. For example, under a condition that a relationship between a position of the target image frame It and the single-directional motion F0→1 is linear, if time points of the image frame I0, the target image frame It, and the image frame I1 are 0, t, and 1, respectively, the scaling module 204 may scale down the single-directional motion F0→1 by t to generate the scaled motion F0-t, wherein “t” is a positive number between 0 and 1, and is a time point at which the target image frame I1 is interpolated between the image frames I0 and I1.
The processing module 206 may be arranged to derive a forward-warped result FWR from a result of performing a forward warping operation and an inverse operation upon the scaled motion F0→t. In detail, the processing module 206 may include a forward warping module 214 and a refining module 216. The forward warping module 214 may be arranged to perform a forward warping operation and an inverse operation upon the scaled motion F0→t, to generate a coarse forward-warped result CFWR, wherein the coarse forward-warped result CFWR includes an alpha map V′t→0 and a forward-warped motion F′t→0, and the alpha map V′t→0 may indicate holes caused by performing the forward warping operation.
The refining module 216 may be arranged to refine the coarse forward-warped result CFWR according to at least the first input image frame F_IIF and the second input image frame S_IIF (e.g. the resized image frames RI0 and RI1), to generate the forward-warped result FWR, wherein the forward-warped result FWR includes a refined alpha map Vt→0 and a refined motion Ft→0. Specifically, the refining module 216 may execute a computer vision algorithm and/or an artificial intelligence (AI) algorithm to refine the alpha map V′t→0 and the forward-warped motion F′t→0 according to at least the first input image frame F_IIF and the second input image frame S_IIF (e.g. the resized image frames RI0 and RI1), respectively, to generate the refined alpha map Vt→0 and the refined motion Ft→0. For example, the above-mentioned holes caused by performing the forward warping operation can be inpainted by the refining module 216.
In this embodiment, in order to enhance the interpolation quality, the backward warping module 212 may be arranged to perform a backward warping operation upon the first input image frame F_IIF (e.g. the resized image frame RI0) according to the forward-warped motion F′t→0 from the forward warping module 214, to generate a backward-warped result I′t→0. The refining module 216 may refine the alpha map V′t→0 and the forward-warped motion F′t→0 according to the first input image frame F_IIF (e.g. the resized image frame RI0), the second input image frame S_IIF (e.g. the resized image frame RI1), and the backward-warped result I′t→0, respectively, to generate the refined alpha map Vt→0 and the refined motion Ft→0.
The inverse module 210 may be arranged to perform an inverse operation upon the forward-warped result FWR, to generate an inversed result IR, wherein the inversed result IR includes an inversed alpha map Vt→1 and an inversed motion Ft→1. Specifically, the inverse module 210 may perform the inverse operation upon the refined alpha map Vt→0 and the refined motion Ft→0, respectively, to generate the inversed alpha map Vt→1 and the inversed motion Ft→1.
The processing module 208 may be arranged to generate the target image frame It according to the image frames I0 and I1, the forward-warped result FWR, and the inversed result IR. In detail, the processing module 208 may include multiple backward warping modules 218 and 220 and a blending module 222. The backward warping module 218 may be arranged to perform a backward warping operation upon the image frame I1 according to the inversed motion Ft→1, to generate a backward-warped result It→1. The backward warping module 220 may be arranged to perform a backward warping operation upon the image frame I0 according to the refined motion Ft→0, to generate a backward-warped result It→0. Afterwards, the blending module 222 may be arranged to blend the backward-warped results It→1 and It→0 according to the refined alpha map Vt→0 and the inversed alpha map Vt→1, to generate the target image frame I1 for performing the frame interpolation between the image frames I0 and I1. Specifically, the blending module 222 may be arranged to multiply the backward-warped result It→0 and the refined alpha map Vt→0 to generate a first multiplication result FMR (e.g. FMR=It→0*Vt→0) multiply the backward-warped result It→1 and the inversed alpha map Vt→1 to generate a second multiplication result SMR (e.g. SMR=It→1*Vt→1) and add the first multiplication result FMR and the second multiplication result SMR to generate the target image frame I1 (e.g. It=FMR+SMR=It→0*Vt→0+It→1*Vt→1).
In Step S300, the single-directional motion F0→1 is obtained by performing motion estimation upon the resized image frames RI0 and RI1 (or the image frames I0 and I1) through the motion estimation module 202, wherein the single-directional motion F0→1 indicates the displacement from the pixels of the image frame I0 to the pixels of the image frame I1, such as 10 in the horizontal direction.
In Step S302, the single-directional motion F0→1 is scaled by the scaling module 204 according to the time point of the target image frame I0.5 to generate the scaled motion F0→0.5. For example, under a condition that a relationship between the position of the target image frame I0.5 and the single-directional motion F0→1 is linear, since the time points of the image frame I0, the target image frame I0.5, and the image frame I1 are 0, 0.5, and 1, respectively, the scaling module 204 may scale down the single-directional motion F0→1 by 2 to generate the scaled motion F0→0.5. In detail, the scaling module 204 may scale down the horizontal position coordinate x and the vertical position coordinate y corresponding to the single-directional motion F0→1 (denoted by F(x, y)) by 2, respectively, to generate the scaled motion F0→0.5. That is, the horizontal position coordinate x and the vertical position coordinate y corresponding to the scaled motion F0→0.5 is F(0.5*x, 0.5*y).
In Step S304, a forward warping operation and an inverse operation are performed upon the scaled motion F0→0.5 by the forward warping module 214, to generate the coarse forward-warped result CFWR, wherein the coarse forward-warped result CFWR includes the alpha map V′0.5→0 and the forward-warped motion F′0.5→0, and the alpha map V′0.5→0 can indicate holes caused by performing the forward warping operation.
In Step S306, the alpha map V′0.5→0 and the forward-warped motion F′0.5→0 are refined by the refining module 216, respectively, to generate the refined alpha map V0.5→0 and the refined motion F0.5→0. In addition, the holes caused by performing the forward warping operation can be inpainted by the refining module 216.
In Step S308, an inverse operation is performed upon the refined alpha map V0.5→0 and the refined motion F0.5→0 by the inverse module 210, respectively, to generate the inversed alpha map V0.5→1 and the inversed motion F0.5→1.
In Step S310, a backward warping operation is performed upon the image frame I1 by the backward warping module 218 according to the inversed motion F0.5→1, to generate a backward-warped result I0.5→1.
In Step S312, a backward warping operation is performed upon the image frame I0 by the backward warping module 220 according to the refined motion F0.5→0, to generate a backward-warped result I0.5→0.
In Step S314, the backward-warped results I0.5→1 and I0.5→0 are blended by the blending module 222 according to the refined alpha map V0.5→0 and the inversed alpha map V0.5→1, to generate the target image frame I0.5 for performing the frame interpolation between the image frames I0 and I1. Specifically, the backward-warped result I0.5→0 and the refined alpha map V0.5→0 are multiplied to generate the first multiplication result FMR (e.g. FMR=I0.5→0*V0.5→0). The backward-warped result I0.5→1 and the inversed alpha map V0.5→1 are multiplied to generate the second multiplication result SMR (e.g. SMR=I0.5→1*V0.5→1). The first multiplication result FMR and the second multiplication result SMR are added to generate the target image frame I0.5 (e.g. I0.5=FMR+SMR=I0.5→0*V0.5→0+I0.5→1*V0.5→1).
After the single-directional motion F1→0 is obtained by the estimation module 202, the scaling module 204 may be arranged to scale the single-directional motion F1→0 according to a time point of the target image frame I1 to generate a scaled motion F1→t. For example, under a condition that a relationship between a position of the target image frame I1 and the single-directional motion F1→0 is linear, if time points of the image frame I0, the target image frame It, and the image frame I1 are 0, t, and 1, respectively, the scaling module 204 may scale down the single-directional motion F1→0 by t to generate the scaled motion F1→t, wherein “t” is a positive number between 0 and 1, and is a time point at which the target image frame It is interpolated between the image frames I0 and I1.
The forward warping module 214 may be arranged to perform a forward warping operation and an inverse operation upon the scaled motion F1→t, to generate a coarse forward-warped result CFWR′, wherein the coarse forward-warped result CFWR′ includes an alpha map V′t→1 and a forward-warped motion F′t→1, and the alpha map V′t→1 may indicate holes caused by performing the forward warping operation. The refining module 216 may be arranged to refine the coarse forward-warped result CFWR′ according to at least the resized image frames RI0 and RI1, to generate a forward-warped result FWR′, wherein the forward-warped result FWR′ includes a refined alpha map Vt→1 and a refined motion Ft→1. Specifically, the refining module 216 may execute a computer vision algorithm and/or an AI algorithm to refine the alpha map V′t→1 and the forward-warped motion F′t→1 according to at least the resized image frames RI0 and RI1, respectively, to generate the refined alpha map Vt→1 and the refined motion Ft→1. For example, the above-mentioned holes caused by performing the forward warping operation can be inpainted by the refining module 216.
In this embodiment, in order to enhance the interpolation quality, the optional backward warping module 212 may be arranged to perform a backward warping operation upon the resized image frame RI1 according to the forward-warped motion F′t→1 from the forward warping module 214, to generate a backward-warped result I′t→1. The refining module 216 may refine the alpha map V′t→1 and the forward-warped motion F′t→1 according to the backward-warped result I′t→1 and the resized image frames Rho and RI1, respectively, to generate the refined alpha map Vt→1 and the refined motion Ft→1.
The inverse module 210 may be arranged to perform an inverse operation upon the forward-warped result FWR′, to generate an inversed result IR′, wherein the inversed result IR′ includes an inversed alpha map Vt→0 and an inversed motion Ft→0. Specifically, the inverse module 210 may perform the inverse operation upon the refined alpha map Vt→1 and the refined motion Ft→1, respectively, to generate the inversed alpha map Vt→0 and the inversed motion Ft→0.
The backward warping module 218 may be arranged to perform a backward warping operation upon the image frame I0 according to the inversed motion Ft→0, to generate a backward-warped result It→0. The backward warping module 220 may be arranged to perform a backward warping operation upon the image frame I1 according to the refined motion Ft→1, to generate a backward-warped result It→1.
Afterwards, the blending module 222 may be arranged to blend the backward-warped results It→0 and It→1 according to the refined alpha map Vt→1 and the inversed alpha map Vt→0, to generate the target image frame It for performing the frame interpolation between the image frames I0 and I1. Specifically, the blending module 222 may be arranged to multiply the backward-warped result It→0 and the inversed alpha map Vt→0 to generate a first multiplication result FMR (e.g. FMR=It→0*Vt→0) multiply the backward-warped result It→1 and the refined alpha map Vt→1 to generate a second multiplication result SMR (e.g. SMR=It→1*Vt→1), and add the first multiplication result FMR and the second multiplication result SMR to generate the target image frame I1 (e.g. It=FMR+SMR=It→0*Vt→0+It→1*Vt→1).
In Step S500, the single-directional motion F1→0 is obtained by performing motion estimation upon the resized image frames RI1 and RI1 (or the image frames I0 and I1) through the motion estimation module 202, wherein the single-directional motion F1→0 indicates the displacement from the pixels of the image frame I1 to the pixels of the image frame I0, such as −10 in the horizontal direction.
In Step S502, the single-directional motion F1→0 is scaled by the scaling module 204 according to the time point of the target image frame I0.5 to generate the scaled motion F1→0.5. For example, under a condition that a relationship between the position of the target image frame I0.5 and the single-directional motion F1→0 is linear, since the time points of the image frame I0, the target image frame It, and the image frame I1 are 0, 0.5, and 1, respectively, the scaling module 204 may scale down the single-directional motion F1→0 by 2 to generate the scaled motion F1→0.5. In detail, the scaling module 204 may scale down the horizontal position coordinate x and the vertical position coordinate y corresponding to the single-directional motion F1→0 (denoted by F(x, y)) by 2, respectively, to generate the scaled motion F1→0.5. That is, the horizontal position coordinate x and the vertical position coordinate y corresponding to the scaled motion F1→0.5 is F(0.5*x, 0.5*Y).
In Step S504, a forward warping operation and an inverse operation are performed upon the scaled motion F1→0.5 by the forward warping module 214, to generate the coarse forward-warped result CFWR′, wherein the coarse forward-warped result CFWR′ includes the alpha map V′0.5→1 and the forward-warped motion F′0.5→1, and the alpha map V′0.5→1 can indicate holes caused by performing the forward warping operation.
In Step S506, the alpha map V′0.5→1 and the forward-warped motion F′0.5→1 are refined by the refining module 216, respectively, to generate the refined alpha map V0.5→1 and the refined motion F0.5→1. In addition, the holes caused by performing the forward warping operation can be inpainted by the refining module 216.
In Step S508, an inverse operation is performed upon the refined alpha map V0.5→1 and the refined motion F0.5→1 by the inverse module 210, respectively, to generate the inversed alpha map V0.5→0 and the inversed motion F0.5→0.
In Step S510, a backward warping operation is performed upon the image frame I0 by the backward warping module 218 according to the inversed motion F0.5→0, to generate a backward-warped result I0.5→0.
In Step S512, a backward warping operation is performed upon the image frame I1 by the backward warping module 220 according to the refined motion F0.5→1, to generate a backward-warped result I0.5→1.
In Step S514, the backward-warped results I0.5→0 and I0.5→1 are blended by the blending module 222 according to the inversed alpha map V0.5→0 and the refined alpha map V0.5→1 to generate the target image frame I0.5 for performing the frame interpolation between the image frames I0 and I1. Specifically, the backward-warped result I0.5→0 and the inversed alpha map V0.5→0 are multiplied to generate the first multiplication result FMR (e.g. FMR=I0.5→0*V0.5→0). The backward-warped result I0.5→1 and the refined alpha map V0.5→1 are multiplied to generate the second multiplication result SMR (e.g. SMR=I0.5→1*V0.5→1). The first multiplication result FMR and the second multiplication result SMR are added to generate the target image frame I0.5 (e.g. I0.5=FMR+SMR=I0.5→0*V0.5→0+I0.5→1*V0.5→1).
In Step S600, the motion estimation is performed upon the first input image frame F_IIF (e.g. the image frame I0 or the resized image frame RI0) and the second input image frame S_IIF (e.g. the image frame I1 or the resized image frame RI1), to obtain the single-directional motion F0→1.
In Step S602, the single-directional motion F0→1 is scaled according to the time point of the target image frame It, to generate the scaled motion F0→t.
In Step S604, the forward-warped result FWR including the refined alpha map Vt→0 and the refined motion Ft→0 is derived from a result of performing a forward warping operation and an inverse operation upon the scaled motion F0→t.
In Step S606, an inverse operation is performed upon the forward-warped result FWR to generate the inversed result IR including the inversed alpha map Vt→1 and the inversed motion Ft→1.
In Step S608, the target image frame It is generated according to the image frame I0, the image frame I1, the forward-warped result FWR, and the inversed result IR.
Since a person skilled in the pertinent art can readily understand details of the steps after reading above paragraphs directed to the frame interpolation module 20 shown in
In summary, by the frame interpolation method of the present invention, under a condition that only a single-directional motion can be obtained due to some reasons (e.g. tradeoff between cost, computation, and/or power saving), frame interpolation can be performed successfully.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
This application claims the benefit of U.S. Provisional Application No. 63/327,898, filed on Apr. 6, 2022. The content of the application is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
20100079667 | Turetken | Apr 2010 | A1 |
Number | Date | Country |
---|---|---|
103702128 | Apr 2014 | CN |
104011771 | Aug 2014 | CN |
103150743 | Apr 2016 | CN |
103702128 | Nov 2016 | CN |
106210767 | Dec 2016 | CN |
111586409 | Aug 2020 | CN |
108886584 | Nov 2020 | CN |
1535296 | May 2016 | TW |
2019084804 | May 2019 | WO |
Number | Date | Country | |
---|---|---|---|
20230328202 A1 | Oct 2023 | US |
Number | Date | Country | |
---|---|---|---|
63327898 | Apr 2022 | US |