1. Field of the Invention
The invention relates to image processing, and in particular, to an optimized image processor structure that reduces occupation of memory bandwidth.
2. Description of the Related Art
a shows a conventional image processing system 100, constructed by a chip and an off-chip memory 110. The chip implements an image processor 120 widely adaptable for various applications such as multimedia players, displays and televisions. An input image #IN may be a static image or a motion picture, transmitted in various signal formats such as National Television System Committee (NTSC) or Phase Alternating Line (PAL) standard compliant signals to the image processing system 100. Generally, the image processor 120 is able to implement sequentially enhancing various aspects of image quality in multiple stages. For example, a first pre-processor 122 may be provided to eliminate cross color interferences within the input image #IN. Cross color is referred to as interferences in chrominance information induced by luminance information. A second pre-processor 124 may be provided to perform noise reduction, and the noise reduction may comprise spatial noise reduction and temporal noise reduction, each using different memory resources. The input image #IN sent to the image processor 120 may be interlaced, which means even lines and odd lines of a frame, each referred to as a field of the frame, are separately and alternatively delivered. Therefore, a de-interlacing process is required to reorganize every two fields into a complete frame. In the image processor 120, a de-interlacer 130 is provided to perform the de-interlacing process based on pre-processed results I(t) output from the second pre-processor 124 to generate a de-interlaced frame P(t). One or more post-processes may be performed after the de-interlacing process, wherein the image data are processed frame by frame. For example, a post-processor 140 may perform motion judder cancellation or image resizing on the de-interlaced frame P(t) output from the de-interlacer 130. Furthermore, frame rate control may also be implemented in a stage (not shown) either prior or posterior to the de-interlacer 130.
To accomplish the aforementioned processes, the image processor 120 must rely on an off-chip memory 110, such as Dynamic Random Access Memory (DRAM) to buffer various intermediate data associated with the input image #IN. Thus, the efficient use of the memory bandwidth between the off-chip memory 110 and image processor 120 is critical to increase performance. For brevity of description, the input image #IN can be denoted as a sequential input field data S(t) where t is a time index. A plurality of field buffers 102 are allocated in the off-chip memory 110 to support the first pre-processor 122, cascaded as a delay line to buffer the field data S(t). The first pre-processor 122 may need one current field data S(t) and two previous field data S(t-1) and S(t-2) to perform a cross color suppression. Thus, at least two I/O transmissions between the field buffers 102 and the first pre-processor 122 are required. Likewise, the second pre-processor 124 requires one previous field data I(t-2) to perform noise reduction on a current field data I′(t). Thus, at least two field buffers 104 are required to buffer a current field data I(t) output from the second pre-processor 124. Meanwhile, the field data I(t) is also sent to a de-interlacer 130 for de-interlacing. The de-interlacer 130 requires two further field buffers 104 to provide previous field data I(t-1) and I(t-2) in order to perform the de-interlacing process to thereby generate a de-interlaced frame P(t). Consequently, a second pre-processor 124 and de-interlacer 130 may jointly require at least four I/O transmissions to access the field buffers 104, in which partial field data such as I(t-2) is redundantly transmitted. The de-interlaced frame P(t) is output to the post-processor 140 such that a post-process can be performed on the current frame P(t) to generate an output image #OUT. The post-process is performed frame by frame, and is not limited to motion judder cancellation (MJC) or image resizing (scaler).
b shows an exemplary de-interlacer 130 and a post-processor 140 of
An exemplary embodiment of an image processing system is provided, comprising an off-chip memory and an image processor. The off-chip memory comprises a plurality of field buffers and frame buffers for buffering intermediate data associated with an input image, and the image processor processes the input image and the intermediate data to generate an output image.
The image processor processes three stages, a pre-processing stage, a de-interlacing stage and a post-processing stage. In the pre-processing stage, the field buffers are read to perform a pre-process, and the pre-processing results are stored in the field buffers. In the de-interlacing stage, a plurality of first line buffers buffer the pre-processed results read from the field buffers, and a de-interlacing process is performed on the pre-processed results to generate de-interlaced results. In the post-processing stage, a post-process is performed on the pre-processed results and the de-interlaced results to generate the output image.
Another embodiment provides an image processor, coupled to an off-chip memory. The image processor comprises a first frame rate controller, receiving the input image and a plurality of buffered field data from a plurality of first field buffers at a first rate, selecting a first number of field data therefrom, and outputting the first number of field data at a second rate. A first pre-processor then performs a pre-process on the first number of field data output from the first frame rate controller to generate an intermediate result. A second pre-processor performs noise reduction on the intermediate result to generate a noise reduction result. A de-interlacer de-interlaces the noise reduction result to generate a de-interlaced result. A first post-processor performs a post-process on the de-interlaced result to generate the output image.
A further embodiment of an image processor is provided, in which, a pre-processor sequentially outputs processed field data to a plurality of field buffers in the off-chip memory at a second rate, and performs noise reduction on the input image based on a previous field data buffered in a particular field buffer. A frame rate controller selects a first number of field data from an output of the pre-processor, and field data buffered in the plurality of field buffers, and outputs the first number of field data at a first rate. Thereafter, a de-interlacer performs a de-interlacing process on the first number of field data to generate a de-interlaced result.
A detailed description is given in the following embodiments with reference to the accompanying drawings.
The invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:
a shows a conventional image processing system;
b shows a de-interlacer 130 and a post-processor 140 according to
a and 4b show embodiments of frame rate control.
The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.
The invention provides an enhanced image processor structure, in which partial data are shared within different stages to reduce memory bandwidth consumption. Specifically, line buffers within a stage may also be shared with another stage, and thereby the usage of field buffers or frame buffers can be reduced. Additionally, a control method is implemented on the field buffers 102 and field buffers 104 to provide an efficient frame rate control mechanism.
As shown in
The first frame rate controller 202 is an optional unit required only when frame rate control is required. The frame rate control can be an up conversion or a down conversion. For example, the first rate may be 50 Hz while the second rate is 60 Hz, or oppositely, the first rate may be 60 Hz while the second rate is 50 Hz. The number of input ports and output ports of the first frame rate controller 202 can be flexibly designed. For example, the delay line of field buffers 102 may comprise more than three field buffers 102. If the first pre-processor 122 requires three field data at the same time, the number of the field buffers 102 must be no less than three, which allows the first frame rate controller 202 to select three field data out of four or more candidates. In the embodiment, the first pre-processor 122 can be a cross color suppressor, and consequently, the pre-process is a cross color suppression process.
Alternatively, the first pre-processor 122 may also be another pre-processor such as a noise reduction unit or an image sharpener. In the embodiment, the pre-process stage may optionally and preferably comprise a second pre-processor 124 following the first pre-processor 122 to perform the pre-processes not performed by the first pre-processor 122. The second pre-processor 124 requires a plurality of field buffers 104 to work, while a de-interlacer 310 in the de-interlacing stage may require the same field data from the field buffers 104. Thus, a plurality of field buffers 104 are allocated in the off-chip memory 210, simultaneously shared by the second pre-processor 124 and the de-interlacing stage. Like the field buffers 102, the field buffers 104 are cascaded as a delay line, dedicated to sequentially buffer pre-processed results I′(T) output from the second pre-processor 124.
In the embodiment, the second pre-processor 124 is a noise reduction unit for performing spatial noise reduction or temporal noise reduction on each field data. As described, the input image #IN are interlaced. Thus, the field data in two consecutive field buffers 104 are associated with different parts of a frame. For example, field buffers of even time indices may be correlated to the top field of a frame, and those of odd time indices may be correlated to even field of the frame. Regarding the case in the second frame rate controller 204, wherein the field buffers 104 sequentially output previous field data I(T-1), I(T-2) and I(T-3) while a current field data I(T) is generated and buffered thereto, the second pre-processor 124 requires a previous field data I(T-2) to perform the noise reduction, and the previous field data I(T-2) is available in a particular field buffer 104 associated with the same part of a frame as the input field data I′(T). Thus, the output of the particular field buffer 104 is connected to the second pre-processor.
In the de-interlacing stage, a de-interlacer 310 performs the de-interlacing process. Generally, three consecutive field data are required for a de-interlacing process. Thus, the de-interlacer 310 may directly receives the current field data I(T) from the second pre-processor 124, and two previous field data I(T-1) and I(T-2) from the field buffers 104. However, in some cases, frame rate control is required. For example, the field data I(T) is provided at a second rate, and the de-interlacer 310 may output a de-interlaced frame P(t) at a first rate. Thus, a second frame rate controller 204 similar to a first frame rate controller 202 can be implemented between the field buffers 104 and the de-interlacer 310. The second frame rate controller 204 receives four inputs, I(T), I(T-1), I(T-2) and I(T-3) from the second pre-processor 124 and field buffers 104, and selects three of them as the three outputs Ia, Ib, Ic that are output at the first rate. To control frame rate, the input ends of the second frame rate controller 204 are designated to be more than its output ends, where the number of output ends is dependent on the number of field data required by the de-interlacer 310.
In
The de-interlacer 310 receives the field data Ia, Ib and Ic line by line, and the field data Ia, Ib and Ic are buffered in a plurality of line buffers 108 before processing.
Regarding the first post-processor 320, likewise, field data are processed line by line. Two line buffers 108 are provided to buffer the de-interlaced frame P(t) output from the de-interlacer core 314. Thus, a current line E0 is sent to the post-processor core 324 while two previous lines E1 and E2 are sent from the corresponding line buffers 108. Meanwhile, the first post-processor 320 receives line data D0, D1 and D2 of the field data Ib from the corresponding line buffers 108 residing in the de-interlacer 310. This approach exhibits the same effects as when the line buffers 108 buffer the P(t-1) shown in the
a and 4b show embodiments of frame rate control. In
b shows another frame rate controller different from the embodiment of
In the embodiment, the pre-process and post-process stages are not limited to comprise one or more processing units. The first and second frame rate controllers 202 and 204 are optional, and can be separately included into or excluded from the embodiment. The frame rate control is not limited to be a down conversion or an up conversion. The number of field buffers 102 and 104 are dependent on requirements during practical implementations, and not necessarily like what is shown in
While the invention has been described by way of example and in terms of preferred embodiment, it is to be understood that the invention is not limited thereto. To the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.