The present invention is in general related to video image processing methods and apparatus and, more particularly, to fast area-selected filtering methods and apparatus that reduce pixel noise and analog artifacts.
Digital video cameras and analog video capturing devices have been widely used in recent years. The ever-greater amount of video images to be processed demand more and more advanced video compression techniques, an example of which is motion estimation that is generally applied to remove data redundancies. Also, during video capturing process, pixel noise and artifacts may be caused by the capturing devices. Thus, image filtering techniques are commonly applied to remove such pixel noise and artifacts.
A video signal comprises temporally consecutive, or sequential, frames of images. Each frame further comprises an array of pixels, which may also be viewed as being arranged in a plurality of horizontal lines. Each pixel typically has three color components: red, green, and blue.
Pixel noise may be described as a random offset in one or more of the color components of the pixels. Therefore, the pixel noise in one pixel of a frame is independent of the pixel noise in another pixel in that same or another frame. In contrast, artifacts are quasi-random offsets exhibiting inter-dependencies between pixels in a line.
In order to remove the pixel noise and artifacts, spatial or temporal filtering methods may be applied. Spatial filtering typically means replacing a pixel with an estimate based on the neighboring pixels thereof, while temporal filtering generally means replacing a pixel with an estimate based on values of that same pixel over time. However, spatial filtering tends to blur sharp edges of the images and therefore reduce the resolution, and temporal filtering may introduce motion blur if objects on the video move at a significant speed. It is therefore desirable to provide improved methods and apparatus for filtering video signals.
Methods and systems consistent with aspects and principles of the present invention may obviate one or more of the above and/or other problems by providing filtering only to selected areas where motions of objects are determined to be slow or still.
Consistent with aspects of the present invention, there is provided a method for processing a video signal having a plurality of sequential frames comprising an array of pixels each having a pixel value and arranged in a plurality of horizontal lines. The method includes receiving a selected frame of the video signal and processing the selected frame, wherein processing the selected frame includes determining an amount of motion of pixels of the selected frame, retaining pixel values of the pixels of the selected frame if the amount of motion in the pixels of the selected frame is greater than a pixel change threshold, and replacing the pixel values of the pixels of the selected frame with filtered pixel values if the amount of motion in the pixels of the selected frame is equal to or less than the pixel change threshold. The method further includes storing the retained pixel values and filtered pixel values of the selected frame.
Consistent with other aspects of the present invention, there is provided a computer system connectable to a video capturing device for receiving a video signal having a plurality of sequential frames comprising an array of pixels each having a pixel value and arranged in a plurality of horizontal lines. The system includes a memory for storing a set of instructions implementing a method for processing the plurality of sequential frames and a processor for executing the set of instructions. The method includes receiving a selected frame of the video signal and processing at least a first pixel of the selected frame, wherein processing at least a first pixel of the selected frame includes determining a first amount of pixel change in the first pixel, wherein the first pixel is in a first line of the selected frame, determining a second amount of pixel change in a second pixel, the second pixel being in a second line of the selected frame and vertically adjacent to the first pixel, determining a third amount of pixel change in a third pixel, the third pixel being in a third line of the selected frame and vertically adjacent to the first pixel, retaining the pixel value of the first pixel if at least two of the first, second, and third amounts of pixel change are greater than a pixel change threshold, and replacing the pixel value of the first pixel with a filtered pixel value if at least two of the first, second, and third amounts of pixel change are equal to or less than the pixel change threshold. The method further includes storing the pixel values and filtered pixel values of the selected frame.
Additional features and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The features and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
The foregoing background and summary are not intended to be comprehensive, but instead serve to help one skilled in the art understand the following implementations consistent with the invention set forth in the appended claims. In addition, the foregoing background and summary are not intended to provide any independent limitations on the claimed invention.
The accompanying drawings show features of implementations consistent with the present invention and, together with the corresponding written description, help explain principles associated with the invention. In the drawings:
The following description refers to the accompanying drawings, in which, in the absence of a contrary representation, the same numbers in different drawings represent similar elements. The implementations set forth in the following description do not represent all implementations consistent with the claimed invention. Instead, they are merely some examples of systems and methods consistent with the invention. Other implementations may be used and structural and procedural changes may be made without departing from the scope of present invention.
Consistent with aspects of the present invention, there is provided an area-selected filtering method and principle that obviate one or more problems of conventional filtering methods. By identifying pixels contained in still or in slow motion portions of an image and applying filtering thereto, the area-selected filtering method avoids possible image blur caused by spatial filtering and motion blur caused by temporal filtering.
In the following description, a video signal comprising progressive image frames is assumed. In other words, spatially consecutive lines of each frame of the video are also displayed consecutively. However, it is to be understood that the area-selected filtering method may also be applied for processing a video signal comprising interlaced images, which shall be apparent to one skilled in the art.
For illustration purposes, it is also assumed that each frame of the video comprises pixels each being represented by the horizontal position and vertical position thereof, wherein the horizontal position is labeled as x and the vertical position is labeled as y. Accordingly, (x, y) represents the pixel at a horizontal distance of x and a vertical distance of y from an origin of the frame.
A video frame at time t=t0 which is currently being filtered is denoted as C and its temporal neighbors are denoted as P and F, wherein P is the previous frame at time t=t0−1 and F is the future frame at time t=t0+1. Thus, C(x,y) represents pixel (x, y) of frame C, etc. The three color components of a pixel may be respectively denoted by a subscript of r, g, or b. For example, Cr(x,y) represents the red color component of pixel C(x,y), and a value of Cr(x,y) indicates the intensity of red color component of pixel C(x,y). In the following description, the color components of each pixel are normalized to the range of 0 to 1. Thus, for example, a maximum value of Cr(x,y) or Cg(x,y) is 1. The composition of all three color components may be represented by a vector, for example, (x,y).
A method for processing a video signal, consistent with the invention, is shown in
However, due to the existence of pixel noise and artifacts, pixel (10, 20) in one frame, e.g., frame C, may not represent the true values of that pixel at time t=t0, in which case a comparison between the values of pixel (10, 20) at frames P, C, F may erroneously indicate that pixel (10, 20) is in a fast or slow motion. To avoid this problem and consistent with the area-selected filtering method, a spatial average of several neighboring pixels, rather than a single pixel, may be compared over several consecutive frames.
For example, a sum of three horizontally neighboring pixels including the pixel current being filtered, (x, y), is first calculated for each of frames P, C, F, as follows:
Each color component of the above sums may be expressed by adding a respective subscript. For example, the red component of F(x,y) may be denoted as
F(x,y),r. Then, each color component of these sums is averaged and the temporal change thereof between frames P, C, F is calculated. A maximum pixel change M(x,y) is defined as follows:
Consistent with the area-selected filtering method, a pixel change threshold u is defined such that if
M(x,y)>u (5)
pixel (x,y) is considered in fast motion at t=t0; otherwise, pixel (x,y) is in slow motion. If M(x,y)=0, pixel (x,y) may be considered still. The value of pixel change threshold u may be any suitable number depending on the particular application. For example, computer simulations show that if u is between 0.1 and 0.25, and preferably 0.15, good-processing results are obtained for many popular applications.
In one aspect, three consecutive horizontal lines may be considered for determining if a pixel is in a fast motion, with the current line in the middle. A criteria for a pixel to be in a fast motion may be set such that two of the three consecutive lines must satisfy inequality (5), given as follows:
N(x,y)=((M(x,y−1)>u)+(M(x,y)>u)+(M(x,y+1)>u))>1, (6)
wherein each of terms (M(x,y−1)>u), (M(x,y)>u), and (M(x,y+1)>u) has a Boolean value of either 0 or 1, and (x, y−1), (x,y), (x, y+1) are vertically neighboring pixels.
One skilled in the art would now understand that N(x,y) indicates a normalized motion of pixel (x, y) at t=t0, where a maximum value of 3 corresponds to a motion of 3√{square root over (2)} pixels per frame time. That is, a value of N(x,y)=3 means that three neighboring lines have changes (i.e., motion) above the threshold. This will occur whenever a 3×3 square of pixels exhibits motion between two frames. Since the greatest distance between pixels in a 3×3 square is 3√{square root over (2)}, this is the amount of motion represented by a value of N(x,y)=3. Because N(x,y) is calculated from M(x,y), which is based on a spatial average of neighboring pixels of (x,y), pixel noise may be effectively suppressed in the representation of motion of pixel (x,y) by N(x,y). Further, because generally the inter-dependency of artifacts does not extend beyond one horizontal line, the criteria given in inequality (6) will effectively suppress artifacts.
Consistent with the present invention, if the amount of motion of the pixel is equal to or below the threshold, the pixel value is retained, at 18. However, if the amount of motion is above the threshold, the pixel value is replaced by a filtered value, at 20. Consistent with the present invention, temporal filtering is applied to filter the slow-motion or still pixels. Thus, image blur caused by spatial filtering may be obviated. Also, because the filtering is only applied to slow-motion or still pixels, motion blur caused in fast-motion pixels may be avoided. In one aspect, the temporal filtering may be implemented by replacing each color component of a pixel that is still or in a slow motion at t=t0 with an estimate thereof, while retaining the pixels determined to be in a fast motion. For example, the estimate may be chosen to be a median value of the respective color components at t=t0−1 t=t0, and t=t0+1.
Assuming a frame size of X pixels in the horizontal direction and Y pixels in the vertical direction, a pseudo-source code may thus be created to reflect the above-described area-selected filtering method:
wherein (x,y), a vector composition of three color components Rr(x,y), Rg(x,y), and Rb(x,y), is the result of filtering pixel
(x,y). After all pixels of the frame have been processed, the retained or replaced pixel values of the frame are stored, at 22.
Computer simulation has been performed using the above code and a simulation result is shown in
As compared to conventional filtering methods, which require computing both the magnitude and direction of the motion of each pixel to estimate the pixel status at a certain moment, the filtering method consistent with the present invention only requires computing the magnitude of pixel motions. Therefore, the area-selected filtering method is much more cost-effective.
Although it was assumed that the video comprises progressive images, one skilled in the art should understand that the above-described filtering method may also be applied to process a video signal comprising interlaced image frames. For example, each frame may be treated as two separate fields, a first field comprising odd lines of the frame and a second field comprising even lines of the frame. The area-selected filtering method may then be applied to the first field and the second field separately. Thus, for interlaced images, inequality (6) becomes inequality (7):
N(x,y)=((M(x,y−2)>u)+(M(x,y)>u)+(M(x,y+2)>u))>1. (7)
In other words, only three consecutive lines in the same field are considered.
Also, in calculating the spatial average of each pixel in formulae (1)˜(3), only three horizontally neighboring pixels were considered. However, one skilled in the art would understand that the number of pixels in calculating the spatial average is not limited to three, but rather in certain applications may be another appropriate value. Clearly, a smaller number of pixels in calculating the spatial average may result in a less satisfactory suppression of pixel noise, while a larger number may cause significant increase in the burden on a processor performing the calculation. Similarly, more or less than three consecutive horizontal lines may be considered in inequality (6) for reducing the artifacts.
Consistent with the present invention, the area-selected filtering method may be implemented in a computer system or in a software that is stored in a computer-readable media. Such implementation is described below with reference to
Referring to
In one aspect, computer system 300 may be implemented as a personal computer, in which case computer system 300 may also include input devices such as a keyboard and a mouse and output devices such as a monitor and speakers (not shown). Processor 310 accordingly may be a central processing unit (CPU). A software program implementing the area-selected filtering method consistent with the present invention may be stored in storage medium 332, such that processor 310 controls the loading of the software program from storage medium 332 to RAM 320 and the execution thereof.
Video images captured by video capturing device 336 may be processed by processor 310 executing instructions contained either in the software program being loaded into RAM 320 or stored in ROM 330. In one aspect, the video images are transmitted from video capturing device 336 at the same time while the images are processed by processor 310. In another aspect, computer system 300 first transfers the video images from video capturing device 336 to storage medium 332, and then loads the video images from storage medium 332 to RAM 320 prior to processing them in processor 310. The results of the video image processing may then be stored in storage medium 32.
In another aspect, computer system 300 may be implemented as an stand-alone image processor, wherein ROM 330 which permanently stores a set of instructions to perform the area-selected filtering method consistent with the present invention. As will be understood by one skilled in the art, other system configurations incorporating processing devices and storage media may be used to execute the methods described above.
The foregoing description of possible implementations consistent with the present invention does not represent a comprehensive list of all such implementations or all variations of the implementations described. The description of only some implementations should not be construed as an intent to exclude other implementations. One skilled in the art will understand how to implement the invention in the appended claims in may other ways, using equivalents and alternatives that do not depart from the scope of the following claims. Moreover, unless indicated to the contrary in the preceding description, none of the components described in the implementations is essential to the invention.