These and other features, objects and advantages of the present invention will become more apparent from the following description when taken in conjunction with the accompanying drawings wherein:
While we have shown and described several embodiments in accordance with our invention, it should be understood that disclosed embodiments are susceptible of changes and modifications without departing from the scope of the invention. Therefore, we do not intend to be bound by the details shown and described herein but intend to cover all such changes and modifications a fall within the ambit of the appended claims.
The preferred embodiments of the present invention will be described in detail hereafter with reference to the attached drawings. Although the embodiments of the present invention can be widely applied to digital broadcasting receiving apparatuses, it is particularly preferable to apply the embodiments of the present invention to apparatuses for receiving terrestrial digital broadcasting toward mobile terminals such as cellular phones (1 segment broadcasting. 1-seg broadcasting for short hereafter). This is because in many cases 1-seg broadcasting sends images after encoding the images at low bit rate such as several hundreds kbps to several Mbps due to limits to the frequency bandwidth of transmitting systems and limits to the processing capacities of mobile terminals and the like. Therefore, apparatuses for receiving and displaying 1-seg broadcasting are under the condition that noises related to encoding conditions and states, that is, block noise and mosquito noise are easily generated, and also due to low bit rate, reproduced images are poor in quality, especially lacking in sharpness. In such apparatuses, the embodiments of the present invention can perform good image quality correction, especially edge enhancement while reducing noises or keeping noises from being emphasized.
Firstly, a first embodiment of the present invention will be described below.
A configuration example of a digital broadcasting receiving apparatus, to which the present invention can be applied, will be described hereafter with reference to
The broadcasting receiving & reproducing circuit 6 is connected to the external antenna 2, and includes a digital tuner 3 for receiving digital broadcasting signals; a video decoding unit 4 for decoding encoded video signals (video signals encoded by means of H.264, for example) out of the received signals received by the digital tuner 3; an audio decoding unit 5 for decoding encoded audio signals (audio signals encoded by means of AAC, for example); and an image processor 100 for performing image quality correction on the video images decoded by the video decoding unit 4. A controller 9 includes, for example, a CPU and sends various control signals related to image quality correction to the image processor 100. And then the image data on which image quality correction has been performed at the image processor 100 is provided to the image output unit 7 for the image to be displayed. The audio data decoded at the audio decoding unit 5 is sent to an audio output unit 8 for the sound of the audio data decoded to be output.
This embodiment is characterized in that image quality correction is performed on an image data using encoding information of an image included in digital broadcasting signals and image information obtained from decoded image data.
The image processor 100 related to this embodiment will be described in detail with reference to
The image processor 100 includes an image input terminal 104 for obtaining an image data from the video decoder unit 4 and an information input terminal 105 for obtaining encoding information. An image data 107 that is input through the image input terminal 104, which includes luminance component data and color-difference component data, is sent to an image quality correction unit 103. Luminance component data 108 of the image data 107 is also sent to a noise detection unit 101. On the other hand, an encoding information data that is input through the information input terminal 105 is also sent to the noise detection unit 101. Encoding information, which is stored in the header of bit streams in digital broadcasting signals, is separated when the coded image is decoded at the video decoding unit 4, and is sent to the information input terminal 105. In this embodiment, encoding information includes bit rate information, quantization step information, DCT coefficient (corresponding to AC component) information, and motion vector information, but any other information can be included in encoding information if necessary.
The noise detection unit 101 detects the locations of noises that appear in images using encoding information that is input through the information input terminal 105 and control signals from the controller 9. In this embodiment, assuming that noise is block noise, block noise generation locations are detected. More specifically, the noise detection unit 101 related to this embodiment specifies which pixel blocks (blocks for short hereafter) include block noise using encoding information such as video bit rate information, quantization step information, DCT coefficient information, motion vector information, and control signals from the controller 9. The locations of block noises detected by the noise detection unit 101 are sent to a setting unit 102 as block noise information. Referring to image information for each block sent from the controller 9, the setting unit 102 sets the quantity of image quality correction for each block of the image. In this embodiment, it is assumed that the quantity of edge enhancement that enhances the edges of an image is set as the quantity of image quality correction. Here the setting unit 102 related to this embodiment changes or modifies the quantity of edge enhancement, which has been set as mentioned above, according to the results detected at the noise detection unit 101. More specifically, when there is no block noise, the quantity of edge enhancement is set maximum and when there is block noise, the quantity of edge enhancement is changed or modified to be lower than maximum so as not to enhance the noise. For example, edge enhancement is not performed on blocks with block noise (i.e., the quantities of edge enhancement are zero) or edge enhancement is performed on blocks with block noise using smaller quantities of edge enhancement than the quantities of edge enhancement for blocks without noise. On the other hand, the quantities of edge enhancement for blocks without noise are set large because there is no noise to be enhanced by large quantities of edge enhancement. The image quality correction unit 103 performs image quality correction including edge enhancement on the image data 107 with the qualities of edge enhancement set at the setting unit 102. Although in the above description it has been described that edge enhancement processing is performed as image quality correction at the image quality correction unit 103, noise canceling processing to reduce noises can be performed instead of edge enhancement processing. For example, noise canceling processing is performed on blocks with block noise. In this case, noise canceling processing can be performed on blocks with block noise using larger quantities of noise canceling than the quantities of noise canceling for blocks without block noise. On the other hand, noise canceling processing is not performed at all on blocks without block noise or noise canceling processing is performed using smaller quantities of noise canceling than the quantities of noise canceling for blocks with block noise. The image data, on which image quality correction is performed in this way at the image quality correction unit 103, is provided to the image output unit 7 through an output image terminal 106.
The whole flow of image quality correction processing at the image processor 100 that is configured in this way will be described with reference to
The functions of the image processor 100 related to this embodiment are not limited by the size of an image size. Therefore these functions of the image processor 100 can be applied to various image display systems. For example, the size of an image of 1-seg broadcasting toward mobile terminals such as cellular phones is the size of QVGA (Quarter Video Graphics Array with 320×240 pixels). The image processor 100 receives a QVGA image from outside and performs image quality correction processing on the QVGA image, and then the image processor 100 outputs the corrected QVGA image. Generally speaking, a QVGA image often gives the impression that the display size of its image is small. Therefore, scaling processing to scale up the QVGA image to a VGA (Video Graphics Array with 640×480 pixels) image can be performed on the QVGA image in parallel with image quality correction processing at the image processor 100 in order to improve the viewability of the displayed QVGA image. The size after scaling processing can be optionally selectable. In the case of a digital TV set such as a PDP-TV set, a LCD-TV set or the like, the size of a processed image can be converted to the size of Hi-Vision TV (with 1920×1088 pixels). As mentioned above, at the image processor 100 related to this embodiment, the size of an input image and the size of an output image can be set optional according to the system to which this embodiment is applied.
Next, each unit of the image processor 100 will be described in detail. Firstly, a noise detection unit 101 will be described in detail with reference to
The bit rate judgment unit 142 compares the bit rate information for a block with a first threshold, that is, bit rate threshold BRth sent from the controller 9, and judges the condition of the block noise for the block. More specifically, when the value of the bit rate obtained from the bit rate information is equal to or lower than the first threshold BRth, the judgment that there is block noise is made and a control signal BRcnt to start up block a noise detection unit 146 is set ON (to enable the block noise detection unit 146). On the other hand, when the value of the bit rate is higher than the first threshold BRth, the judgment that there is no block noise is made and the control signal BRcnt is set OFF (to disable the block noise detection unit 146). The control signal BRcnt set in this way is sent to the block noise detection unit 146.
Here how to set the first threshold, that is, the bit rate threshold BRth, will be described. An image with a higher bit rate has higher quality, and an image with a lower bit rate more often loses its original image information, resulting in the degradation of the image quality and frequent block noise occurrence.
As is clear from
The video bit rate threshold BRth can be changed according to the types (genres) of digital broadcasting programs that are input to the digital broadcasting receiving apparatus 1. Here the types of digital broadcasting programs mean the categories of image contents of, for example, drams, sports, news, movies. For example, assuming that a threshold for dramas is the reference video bit rate threshold BRth, a threshold for sports programs with fast-moving scenes can be set lower than the threshold BRth and a threshold for news with comparatively slow-moving scenes can be set higher than the threshold BRth. A threshold for movies can be equal to or a little lower than the threshold BRth.
The quantization step judgment unit 143 compares the quantization step information for a block with a second threshold, that is, quantization step threshold Qth sent from the controller 9, and judges the condition of the block noise for the block. More specifically, when the value of the quantization step obtained from the quantization step information is equal to or larger than the second threshold Qth, the judgment that there is block noise is made and a control signal Qcnt to start up the block noise detection unit 146 is set ON (to enable the block noise detection unit 146). On the other hand, when the value of the quantization step is smaller than the second threshold Qth, the judgment that there is no block noise is made and a control signal Qcnt is set OFF (to disable the block noise detection unit 146). The control signal Qcnt set in this way is sent to the block noise detection unit 146.
Here how to set the second threshold, that is, the quantization step threshold Qth, will be described. When an image is encoded, quantization step is used to quantize the image data for a block that has been transformed by two-dimensional DCT. If the value of quantization step is set larger, compression ratio becomes larger, and higher encoding efficiency can be achieved. However, if the value of quantization step is set larger, the encoded image data more often loses its original image information, resulting in the degradation of its image quality and frequent block noise occurrence.
As is clear from
The DCT coefficient judgment unit 144 compares the DCT coefficient information for a block with the third threshold, that is, DCT coefficient threshold Dth sent from the controller 9, and judges the condition of the block noise for the block. More specifically, when the number of zeroes in the two-dimensional DCT coefficients (corresponding to AC components) obtained from the DCT coefficient information is equal to or larger than the third threshold Dth, the judgment that there is block noise is made and a control signal Dcnt to start up the block noise detection unit 146 is set ON (to enable the block noise detection unit 146). On the other hand, when the number of zeroes in the two-dimensional DCT coefficients (corresponding to AC components) is smaller than the third threshold Dth, the judgment that there is no block noise is made and a control signal Dcnt is set OFF (to disable the block noise detection unit 146). The control signal Dcnt set in this way is sent to the block noise detection unit 146.
Here how to set the third threshold, that is, DCT coefficient threshold Dth, will be described. As mentioned above, the two-dimensional DCT coefficients consist of DC component that shows the image component with the lowest spatial frequency (the first low frequency term) and plural AC components that show the image components with higher spatial frequencies (higher frequency terms) in a block. Among these coefficients, it is AC components that are referred to at the DCT coefficient judgment unit 144. High frequency terms in the DCT coefficients can be intentionally dropped (AC components can be set zero) by setting the value of quantization step large with the result that the encoding efficiency is improved. However lack of high frequency terms reduces fineness and sharpness of an image, resulting in frequent block noise occurrence.
As is clear from
The motion vector judgment unit 145 compares the motion vector information for a block with a fourth threshold, that is, motion vector threshold MVth sent from the controller 9, and judges the condition of the block noise for the block. More specifically, when the value of motion vector obtained from the motion vector information is equal to or larger than the fourth threshold MVth, the judgment that there is block noise is made and a control signal MVcnt to start up the block noise detection unit 146 is set ON (to enable the block noise detection unit 146). On the other hand, when the value of motion vector is smaller than the fourth threshold MVth, the judgment that there is no block noise is made and a control signal MVcnt is set OFF (to disable the block noise detection unit 146). The control signal MVcnt set in this way is sent to the block noise detection unit 146.
Here how to set the fourth threshold, that is, the motion vector threshold MVth, will be described. A motion vector is one of the parameters that utilizes the fact that there is a high correlation between two successive images, and the motion vector is information that shows the relative position between an encoding target block and a reference block. A quantity of motion vector is a value that shows the distance between the coordinate positions of two blocks, and more specifically it is indicated in the number of pixels. The larger the motion of an image becomes, the more the number of encoding target blocks increases and also the larger the quantity of motion vector for each block becomes, resulting in the increase of the amount of code generation. However, in general, the limitations of system resources and the like impose restrictions on the maximum amount of code generation. The restrictions on the amount of code generation result in frequent block noise occurrence.
As is clear from
As mentioned above, each judgment unit of 142 to 145 obtains the corresponding encoding information, compares it with the corresponding threshold and makes the judgment whether a reference block has block noise or not. Then each judgment unit sends its judgment result to the block noise detection unit 146 after setting corresponding control signal BRcnt, Qcnt, Dcnt or MVcnt ON or OFF.
The block noise detection unit 146 makes the judgment whether the reference block has block noise or not using these control signals BRcnt, Qcnt, Dcnt, and MVcnt. In other words, the block noise detection unit 146 specifies blocks where block noise exists using the thresholds. For example, the block noise detection unit 146 makes the judgment that a block has block noise if any one of control signals BRcnt, Qcnt, Dcnt, or MVcnt is ON. If all the control signals are OFF, the judgment that there is no block noise in the block is made. In this way the block noise detection unit 146 determines whether there is block noise or not for each block and sends the result to the setting unit 102 through an output terminal 147.
Firstly, at Step 150, the noise detection unit 101 obtains the luminance components of decoded image data and encoding information of the image data. Here as mentioned above, encoding information 105 includes video bit rate information, quantization step information, DCT coefficient (corresponding to AC component) information, and motion vector information. Secondly, at Step 151, the bit rate judgment unit 142 compares bit rate information with the first threshold BRth. When bit rate information is equal to or lower than BRth (in the case of yes), the control signal BRcnt is set ON and the flow proceeds to Step 155.
At above-mentioned Step 151, if the result of the judgment is “no”, the control signal BRcnt is set OFF and the flow proceeds to Step 152. Thirdly, at Step 152, the quantization step judgment unit 143 compares quantization step information with the second threshold Qth. When quantization step information is equal to or larger than Qth (in the case of yes), the control signal Qcnt is set ON and the flow proceeds to Step 155.
At above-mentioned Step 152, if the result of the judgment is “no”, the control signal Qcnt is set OFF and the flow proceeds to Step 153. Then, at Step 153, the DCT coefficient judgment unit 144 compares DCT coefficient information (the number of zeroes in DCT coefficients corresponding to AC components) with the third threshold Dth. When DCT coefficient information is equal to or larger than Dth (in the case of yes), the control signal Dcnt is set ON and the flow proceeds to Step 155.
At above-mentioned Step 153, if the result of the judgment is “no”, the control signal Dcnt is set OFF and the flow proceeds to Step 154. Lastly, at Step 154, the motion vector coefficient judgment unit 145 compares motion vector information with the fourth threshold MVth. When motion vector information is equal to or larger than MVth (in the case of yes), the control signal MVcnt is set ON and the flow proceeds to Step 155. On the other hand, at Step 15, if the result of the judgment is “no”, the control signal MVcnt is set OFF and the flow proceeds to Step 156.
Step 155 and Step 156 are operations at the block noise detection unit 146. More specifically, if anyone of the judgment results at Step 151 to 154 is “yes”, that is, if any one of control signals BRcnt, Qcnt, Dcnt, or MVcnt is “ON”, the judgment that the block has block noise is made at step 155, and the flow ends. On the other hand, if all the judgment results at Step 151 to 154 is “no”, that is, if all the control signals BRcnt, Qcnt, Dcnt, and MVcnt are “OFF”, the judgment that the block has no block noise is made at step 156, and the flow ends.
In the operation flow, although the judgment that a block has block noise is made if any one of the judgment results at Step 151 to 154 is “yes”, how to make the judgment is not limited to the way. For example, the judgment that a block has block noise can be made if any or predetermined two or three of the four control signals are “ON”.
This embodiment specifies which blocks include block noise in this way. And edge enhancement is not performed for blocks with block noise while edge enhancement is performed on blocks without block noise. In other words, in this embodiment, blocks without block noise are blocks targeted for image quality correction.
a is an explanatory diagram showing an example of a block noise generation state of an input image 160. In the input image 160, let's suppose that blocks filled with wave lines (161 and the like) show blocks with block noise, and blocks with blank (162 and the like) show blocks without block noise.
The setting unit 102 will be described in detail with reference to
The term “block” here is the unit of the pixel size that is a target for motion compensation processing at image encoding. Motion compensation processing is a technique that effectively encodes image data using the results of the examination of changes between two images that exist in two frames. A pixel size is the number of pixels that constitute a block. A pixel size is usually indicated by M×N, that is, the block is made up with the pixels arranged in a rectangle. For example, the pixel size of a block used in MPEG-1 or MPEG-2 is fixed at 16×16 pixels. In MPEG-4, both a block with 16×16 pixels and a block with 8×8 pixels can be used. And in H.264, a block with 16×16 pixels, a block with 16×8 pixels, a block with 8×16 pixels, and a block with 8×8 pixels can be used. A block with 8×8 pixels can be specified that it be divided into 4 types of subblocks with 8×8 pixels, 8×4 pixels, 4×8 pixels, or 4×4 pixels. In
In this embodiment, when image quality correction is performed on a block 171 in an input image (luminance component) 170 (under the assumption that the pixel size of the block 171 is 4×4 pixels), it is checked whether a pixel 172 to a pixel 175 other than outer circumferential pixels in the block 171 include high frequency components or not. Hereafter a parameter to indicate whether high frequency components are included or not shall be called pixel state coefficient X. Pixel state coefficient X is derived from the calculation that refers to the values of at least two pixels. In the example of
Xh=|A−B|+″C−D| (Eq. 1)
Xh=|A−C|+|B−D| (Eq. 2)
Here A, B, C, and D in Eq. 1 and Eq. 2 are luminance signal levels, or high frequency component levels in luminance signals at pixel A 172, pixel B 173, pixel C 174, and pixel D 175 respectively.
The calculation of pixel state coefficient along the horizontal direction Xh and pixel state coefficient along the vertical direction Xv are performed, for example, at the controller 9 in
In order to implement the characteristics shown by the dashed line 180 or the solid line 181 in
The setting unit 102 determines the final quantity of edge enhancement for each block using the quantity of edge enhancement derived from the edge enhancement table and the block noise information sent from the noise detection unit 101 (the block noise detection unit 146). How to determine the final quantity of edge enhancement will be described with reference to
The setting unit 102 related to this embodiment is equipped with a memory that is not shown in the figure to temporarily store the block noise information sent from the noise detection unit 101 (the block noise detection unit 146) and the quantity of edge enhancement derived from the edge enhancement table. This memory is equipped with a first memory area to store block noise information as shown in
When the block noise information sent from the noise detection unit 101 (block noise detection unit 146) is input to the setting unit 102, the block noise information is stored in the address for the block corresponding to the noise information in the first memory area. On the other hand, the quantity of edge enhancement derived from the edge enhancement table is stored in the address for the block corresponding to the quantity of edge enhancement in the second memory area.
Here if the block noise information stored in an address of the first memory area shows “There is block noise”, “0” is stored in the corresponding address of the second memory area. For example, as shown in
On the other hand, if the block noise information stored in an address of the first memory area shows “There is no block noise”, the quantity of edge enhancement derived from the edge enhancement table is stored in the corresponding address of the second memory area. For example, as shown in
In the example, although the content of the addresses in
Here the description will be made under the assumption that each block of an input image consists of 4×4 pixels. Firstly, at Step 190, the judgment whether a reference block has block noise or not is made with reference to block noise information sent from the noise detection unit 101. As a result, if the block has block noise, the flow proceeds to Step 197, where “0” is stored in the memory (in the second memory area) as the quantity of edge enhancement, and then the flow proceeds to Step 196. At the same time, information that the block has block noise is stored in the first memory area.
On the other hand, if the judgment that the block has no block noise is made, the flow proceeds to Step 191, where the controller 9 obtains the image data (4×4 pixels) of the reference block. At the same time, information that the block has no block noise is stored in the first memory area. Then after referring to the pixel values of four pixels located at the center of the block at Step 192, the controller 9 calculates pixel state coefficient X and sends the calculation result to the setting unit 102 at Step 193. Next, at Step 194, the setting unit 102 obtains the quantity of edge enhancement EM corresponding to the block from the table based on the pixel state coefficient X. Then, at Step 195, the obtained quantity of edge enhancement EM is stored in the corresponding address of the block in the second memory area. The flow proceeds Step 196 afterward. At Step 196, the judgment whether there is the next block to be referred to or not is made and if there is not the next block to be referred to, these successive processes stop. If there is the next block to be referred to, the flow return to Step 190 and these successive processes are repeated until there is no block to be referred to (,that is, until decoding of all the blocks ends).
The quantities of edge enhancement EM (or “0”) obtained in this way are sent to the image quality correction unit 103. Then the image quality correction unit 103 performs edge enhancement on each block using the corresponding quantity of edge enhancement EM (or “0”).
As described above, this embodiment determines the quantity of image quality correction for each block using block noise information and image information related to the block. Therefore, this embodiment can perform more accurate image quality correction. In the embodiment of the present invention, although edge enhancement processing has been used as an example of image quality correction to describe the present invention, image quality correction is not limited to edge enhancement processing. For example, noise reduction processing can also be applied to this embodiment as an example of image quality correction. In the case of noise reduction processing, as contrasted with edge enhancement processing, noise reduction processing is performed when there is block noise and noise reduction processing is not performed or reduction processing is performed with a small quantity of noise reduction when there is no block noise. In this case, it will be understood that noise reduction processing can be also performed with reference to image information related to a block. For example, even if there is block noise, noise reduction processing can be performed with a smaller quantity of noise reduction when there are a large number of high frequency components in the block and with a larger quantity of noise reduction when there are a small number of high frequency components in the block.
Next, a second embodiment of the present invention will be described below. In the first embodiment of the present invention, the set quantities of edge enhancement have been applied to all pixels in a block. As contrasted with the first embodiment, the second embodiment of the present invention changes a quantity of edge enhancement for a pixel according to the location of the pixel in the block. More specifically, the quantity of edge enhancement to be given to each pixel is set with reference to the quantities of blocks lying adjacent to the block. This method will be described in detail with reference to
EMa=(1/4×EMs0)+(3/4×EMs1) (Eq. 3)
EMb=EMs1 (Eq. 4)
EMc=EMs1 (Eq. 5)
EMd=(1/4×EMs2)+(3/4×EMs1) (Eq. 6)
A symbol 201 in
EMe=(1/4×EMs3)+(3/4×EMs4) (eq. 7)
EMf=EMs4 (Eq. 8)
EMg=EMs4 (Eq. 9)
EMh=(1/4×EMs5)+(3/4×EMs4) (Eq. 10)
A symbol 211 in
This embodiment enables finer image quality correction because the quantity of edge enhancement can be set for each pixel in a block, not for the whole block.
The image processor 100 can determine the quantity of image quality correction based on encoding information and category information of a program, not based on encoding information and image information of blocks. In other words, it means that the quantity of image quality correction that has been determined based on encoding information, for example, in the first embodiment of the present invention can be modified based on category information of a program. For example, in the case that image quality correction is edge enhancement, if the program is a sport program, the quantity of edge enhancement can be set larger than the quantity determined based on encoding information, and if the program is a news program, the quantity of edge enhancement can be set smaller than the quantity determined based on encoding information.
The present invention can be applied, for example, to a note PC or a desktop PC that is equipped with a receiving & reproducing function of a digital broadcasting such as 1-seg broadcasting, an apparatus that is equipped with an image reproducing function such as a digital TV set, a car navigation system, a potable DVD player or the like.
Number | Date | Country | Kind |
---|---|---|---|
2006-101400 | Apr 2006 | JP | national |