The above and other aspects of the present invention will become more apparent by describing in detail an exemplary embodiment thereof with reference to the attached drawings in which:
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The image resolution conversion apparatus 300 includes an initial interpolation unit 310, a motion estimation unit 320, a motion outlier detection unit 330, an edge detection unit 340, a directional function generation unit 350, and a POCS reconstruction unit 360.
The initial interpolation unit 310 initially interpolates an input low-resolution image frame y(m1,m2,k) into a high-resolution image frame x(n1,n2,k). Initial interpolation may be bilinear interpolation or bicubic interpolation, which is well known to those of ordinary skill in the art and thus will not be described here.
The motion estimation unit 320 performs motion estimation on a kth initially interpolated high-resolution image frame x(n1,n2,k) at a time tr in order to predict a motion vector u=(u,v). A motion estimation algorithm may be performed using block-based motion estimation, pixel-based motion estimation, or a robust optical flow algorithm. Since block-based motion estimation has problems such as motion prediction errors and block distortion, pixel-based motion estimation and the robust optical flow algorithm are used for motion estimation in an exemplary embodiment of the present invention.
The robust optical flow algorithm predicts a motion vector using a motion outlier. The motion outlier can be classified into an outlier with respect to data preservation constraints and an outlier with respect to spatial coherence constraints. In general, a region having a large amount of motion is detected as the outlier with respect to the data preservation constraints, and an edge portion of an image frame or a region having a sharp change in a pixel value is detected as the outlier with respect to the spatial coherence constraints.
An outlier map ME
An outlier map ME
where outlierE
Block-based motion estimation, pixel-based motion estimation, the robust optical flow algorithm are well known to those of ordinary skill in the art and thus will not be described here.
The motion outlier detection unit 330 detects pixels having a large amount of motion prediction errors based on motion information estimated by the motion estimation unit 320 in order to generate a motion outlier map M(m1,m2,k).
The motion outlier map M(m1,m2,k) obtained by the motion outlier detection unit 330 is expressed as follows.
M(m1,m2,k)=D(ME
where D (.) indicates down sampling with respect to horizontal and vertical directions.
The edge detection unit 340 detects an edge from the input low-resolution image frame y(m1,m2,k) in order to generate an edge map E(m1, m2,k) and detects the direction of the edge in order to generate edge direction information θe.
The generation of the edge map E(m1,m2,k) is performed as follows. The edge detection unit 340 defines a region having larger gradients with respect to horizontal and vertical directions than a predetermined threshold ThE in the low-resolution image frame y(m1,m2,k) as an edge region and defines the other regions as non-edge regions.
A region corresponding to E(m1,m2,k)=1 means an edge region and a region corresponding to E(m1,m2,k)=O means a non-edge region.
As illustrated in
and the vertical side of the triangle is a vertical change rate of the low-resolution image frame y, i.e.,
the oblique side of the triangle is
In this triangle, the included angle θe between the oblique side and the horizontal side is an edge direction and is calculated as follows. In other words, the edge detection unit 340 generates edge direction information θe by calculating Equation (8).
The directional function generation unit 350 generates a directional point spread function based on the generated edge map and edge direction.
More specifically, the directional function generation unit 350 generates a colinear Gaussian function like Equation (9) for a pixel corresponding to E(m1,m2,k)=O, i.e., a pixel in a non-edge region.
In
As such, for pixels in a non-edge region, Gaussian functions having the same shape are generated regardless of directivities.
The directional function generation unit 350 generates a one-dimensional Gaussian function like Equation (10) for a pixel corresponding to E(m1, m2,k)=1, i.e., a pixel in an edge region, based on edge direction information.
where ne means a distance from a central pixel. In other words, ne at the central pixel is 0 and ne at a pixel located 1 pixel from the central pixel is 1.
Since function values of pixels decrease as distances of the pixels from the center increase in the one-dimensional Gaussian function, weights applied to the pixels decrease as distances of the pixels from the center increase.
Referring to
To sum up, the directional function generation unit 350 generates a directional point spread function ĥt
The directional point spread function ĥt
The POCS reconstruction unit 360 improves the resolution of an image using the low-resolution image frame y(m1,m2,k), the initially interpolated high-resolution image frame x(n1,n2,k), the motion vector u=(u,v), the outlier map M(m1,m2,k) and the directional point spread function ĥt
In other words, the POCS reconstruction unit 360 calculates a residual term as in Equation (12) by substituting Equation (II) into Equation (1) and generates the convex set Ct
Finally, the super-resolution image frame {circumflex over (x)}(n1,n2,tr) is obtained as in Equation (13) by substituting Equation (11) into Equation (3).
The operation and configuration of the POCS reconstruction unit 360 are well known to those of ordinary skill in the art and thus will not be described here. However, in an exemplary embodiment of the present invention, the POCS reconstruction unit 360 reduces incorrect compensation by excluding pixels having a large amount of motion prediction errors from a resolution conversion process based on the motion outlier map M(m1,m2,k) generated by the motion outlier detection unit 330. In other words, for the pixels having a large amount of motion prediction errors, the iteration unit 136 of
In operation 802, the input low-resolution image frame y(m1,m2,k) is initially interpolated into the high-resolution image frame x(n1,n2,tr).
In operation 804, motion of the initially interpolated high-resolution image frame x(n1, n2,k) is estimated in order to predict the motion vector u=(u,v).
In operation 806, pixels having a large amount of motion prediction errors are detected based on the estimated motion information in order to generate the motion outlier map M(m1,m2,k).
In operation 808, an edge is detected from the input low-resolution image frame y(m1,m2,k), the direction of the detected edge is detected, and the edge map E(m1,m2,k) and the edge direction information θe are generated.
In operation 810, the directional point spread function is generated based on the generated edge map E(m1, m2,k) and edge direction information θe.
In operation 812, a difference between motions of the low-resolution image frame y(m1,m2,k) and the initially interpolated high-resolution image frame x(n1,n2,tr) is corrected using the motion vector u=(u,v).
In operation 814, the residual term is generated using the low-resolution image frame y(m1, m2,k) and the high-resolution image frame x(n1,n2,tr) whose motions are corrected and using the directional point spread function ĥt
In operation 816, the convex set Ct
In operation 818, the initially interpolated high-resolution image frame x(n1,n2,tr) is renewed based on the motion outlier map (m1, m2,k) and whether or not the condition for the convex set Ct
More specifically, if the condition for the convex set Ct
In operation 820, if the condition for the convex set Ct
Meanwhile, an exemplary embodiment of the present invention can be embodied as a program that can be implemented on computers and can be implemented on general-purpose digital computers executing the program using recording media that can be read by computers.
Examples of the recording media include magnetic storage media such as read-only memory (ROM), floppy disks, and hard disks, optical data storage devices such as CD-ROMs and digital versatile discs (DVD), and carrier waves such as transmission over the Internet.
According to exemplary embodiments of the present invention, by using an appropriate point spread function corresponding to the direction of a detected edge, it is possible to improve resolution while maintaining the edge.
While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims.
Number | Date | Country | Kind |
---|---|---|---|
10-2006-0054375 | Jun 2006 | KR | national |