Patent Application
20210287339
The present invention relates to an image processing technique, and more particularly, to an image processing apparatus using an artificial intelligence super-resolution circuit and a follow-up resizer circuit for image resizing.
Modern image display devices typically enhance the images before they are displayed. Some devices can perform super-resolution for image resizing. For example, a super-resolution operation can be used to up-scale a low resolution image (e.g., an image with 720×480 pixels) to a high resolution image (e.g., an image with 3840×2160 pixels). The super-resolution function is a key feature on a television and a smartphone. However, a typical edge device, such as the television or the smartphone, has limited computing power due to strict requirements on power consumption, thermal performance and hardware cost. A super-resolution function with image enhancement may have limited scaling capability due to limited computing power of the typical edge device. Thus, there is a need for an innovative design that has a super-resolution function integrated in an image processing pipeline and is capable of providing arbitrary-ratio image resizing.
One of the objectives of the claimed invention is to provide an image processing apparatus using an artificial intelligence super-resolution circuit and a follow-up resizer circuit for image resizing.
According to a first aspect of the present invention, an exemplary image processing apparatus is disclosed. The exemplary image processing apparatus includes a super-resolution (SR) circuit and a resizer circuit. The SR circuit is arranged to perform an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, and the SR operation is based, at least in part, on one or more artificial intelligence (AI) models. The resizer circuit is arranged to perform a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and no AI model is involved in the resize operation.
According to a second aspect of the present invention, an exemplary image processing apparatus is disclosed. The exemplary image processing apparatus includes a super-resolution (SR) circuit and a resizer circuit. The SR circuit is arranged to perform an SR operation upon a first image to generate a second image, wherein a resolution of the second image is not lower than a resolution of the first image, the SR circuit supports fixed scaling ratios only, and any scaling ratio employed by the SR operation is selected from the fixed scaling ratios. The resizer circuit is arranged to perform a resize operation upon the second image to generate a third image, wherein a resolution of the third image is not lower than the resolution of the second image, and any scaling ratio employed by the resize operation is an arbitrary scaling ratio supported by the resizer circuit.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the following description and claims, which refer to particular components. As one skilled in the art will appreciate, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not in function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
The SR circuit 102 is arranged to perform an SR operation upon the pre-processed image IMG_PP to generate an SR image IMG_SR, wherein a resolution of the SR image IMG_SR is not lower than a resolution of the pre-processed image IMG_PP. In a case where a first scaling ratio S1 (S1=1×) is employed by the SR operation, the resolution of the SR image IMG_SR is equal to the resolution of the pre-processed image IMG_PP. For example, the SR image IMG_SR and the pre-processed image IMG_PP may have the same number of pixels for the same display size. In another case where a second scaling ratio S2 (S2>1×) is employed by the SR operation, the resolution of the SR image IMG_SR is higher than the resolution of the pre-processed image IMG_PP. For example, the SR image IMG_SR may have A pixels and the pre-processed image IMG_PP may have B pixels for the same display size, where A is larger than B.
In this embodiment, the SR circuit 102 performs the SR operation with aid of artificial intelligence (AI). Specifically, the SR operation is based, at least in part, on one or more AI models. Hence, the SR circuit 102 can perform the SR operation with image enhancement. Examples of AI models include, but are not limited to, convolutional neural network (CNN) models, machine learning models, deep learning models, etc. Furthermore, the SR circuit 102 may be implemented by an AI processing unit (APU), a graphic processing unit (GPU), an application-specific processor, or a general-purpose processor.
The resizer circuit 104 is arranged to perform a resize operation upon the SR image IMG_SR to generate an output image IMG_OUT, wherein a resolution of the output image IMG_OUT is not lower than the resolution of the SR image IMG_SR. For example, the resize operation with a scaling ratio larger than 1 is performed to up-scale a lower resolution image to a higher resolution image, such that the resolution of the output image IMG_OUT is higher than the resolution of the SR image IMG_SR, that is, the output image IMG_OUT may have C pixels and the SR image IMG_SR may have D pixels for the same display size, where C is larger than D. For another example, the resolution of the SR image IMG_SR is equal to a target resolution, and the resize operation with a scaling ratio equal to 1 is applied to the SR image IMG_SR, such that the resolution of the output image IMG_OUT is equal to the resolution of the SR image IMG_SR, that is, the output image IMG_OUT and the SR image IMG_SR may have the same number of pixels for the same display size. In this embodiment, the resizer circuit 104 performs the resize operation without aid of AI, such that no AI model is involved in the resize operation.
The result collection buffer 108 is arranged to buffer pixels of the output image IMG_OUT, and output buffered pixels of the output image IMG_OUT via an output port P6 of the image processing apparatus 100.
Since the SR circuit 102 is designed to perform artificial intelligence super-resolution (AISR), no arbitrary-ratio scaling can be achieved by the SR circuit 102. For example, one set of trained CNN parameters is required by the SR operation for applying one scaling ratio, and the AI engine (e.g., SR circuit 102) can acquire only a limited number of trained CNN parameter sets. Hence, the SR circuit 102 supports fixed scaling ratios only, and any scaling ratio employed by the SR operation is selected from the fixed scaling ratios.
In contrast to the SR circuit 102, the resizer circuit 104 is designed to perform the resize operation without aid of AI. Hence, arbitrary-ratio scaling can be achieved by the resizer circuit 104. That is, any scaling ratio employed by the resize operation is an arbitrary scaling ratio supported by the resizer circuit 104.
As mentioned above, the SR circuit 102 performs the SR operation with AI-aided image enhancement, but lacks scaling flexibility due to fixed-ratio scaling; and the resize circuit 104 performs the arbitrary-ratio resize operation without AI-aided image enhancement. With the help of the resize circuit 104 cascaded to the SR circuit 102, arbitrary-ratio scaling with AI-aided image enhancement can be achieved.
In a second case where the resolution W×H of the pre-processed image IMG_PP is higher than a high definition (HD) resolution but not higher than the FHD resolution, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 2× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programmed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.
In a third case where the resolution W×H of the pre-processed image IMG_PP is higher than 960×540 but not higher than the HD resolution, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 3× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programmed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.
In a fourth case where the resolution W×H of the pre-processed image IMG_PP is not higher than 960×540, the SR circuit 102 performs the fixed-ratio SR operation with the fixed scaling ratio 4× upon the pre-processed image IMG_PP, and then the resizer circuit 104 refers to discrepancy between a resolution of the pre-processed image IMG_PP and the target UHD resolution to program a scaling ratio adaptively, and performs the arbitrary-ratio resize operation with the adaptively programmed scaling ratio upon the SR image IMG_SR to generate the output image IMG_OUT with the target UHD resolution.
As illustrated in
When the image processing apparatus 100 is employed by an edge device such as a television or a smartphone, the SR circuit 102 is an AI engine with limited computing power due to strict requirements on power consumption, thermal performance and hardware cost. In this embodiment, the SR circuit 102 is cascaded to the image pre-processing circuit 106, and the image pre-processing circuit 106 applies image enhancement (e.g., NR) to the input image IMG_IN before AISR. Since the SR circuit 102 applies AISR to the pre-processed image IMG_PP with enhanced image quality, the image quality of the SR image IMG_SR can be improved.
It should be noted that the image pre-processing circuit 106 may be optional. In an alternative design, the image processing apparatus 100 may be modified to omit the image pre-processing circuit 106, and the SR circuit 102 is modified to perform the SR operation upon the input image IMG_IN to generate the SR image IMG_SR. In practice, any image processing apparatus using a fixed-ratio SR circuit (e.g., an SR circuit with aid of AI) and an arbitrary-ratio resizer circuit (e.g., a resizer circuit without aid of AI) to achieve arbitrary-ratio scaling with image enhancement (e.g., arbitrary-ratio scaling with AI-aided image enhancement) falls within the scope of the present invention.
In some embodiments of the present invention, one or both of the image pre-processing circuit 106 and the SR circuit 102 may employ raster scan based processing for making the image processing apparatus 100 have low latency and low bandwidth. For example, in the embodiment shown in
Consider a case where the image IMG is an input image to be processed by one image processing pipeline stage. In accordance with a raster scan order R, pixels of the same pixel line are sequentially processed from the left-most pixel P1 to the right-most pixel PM, and pixel lines of the same image are sequentially processed from the top-most pixel line L1 to the bottom-most pixel line LN.
Consider another case where the image IMG is an output image generated from one image processing pipeline stage. In accordance with the raster scan order R, pixels of the same pixel line are sequentially output from the left-most pixel P1 to the right-most pixel PM, and pixel lines of the same image are sequentially output from the top-most pixel line L1 to the bottom-most pixel line LN.
When the image IMG shown in
When the image IMG shown in
Due to the inherent characteristic of the raster scan based processing, an output port P4 of the SR circuit 102 can be directly linked to an input port P5 of the resizer circuit 104, that is, the image processing apparatus 100 has no frame buffer (which is used to store one full image) or block buffer (which is used to store an m×n block in an image, where the m×n block has m pixels in a width direction of the image and n pixels in a height direction of the image, and n is larger than 1) coupled between the output port P4 of the SR circuit 104 and the input port P5 of the resizer circuit 104. Similarly, an output port P2 of the image pre-processing circuit 106 can be directly linked to an input port P3 of the SR circuit 102, that is, the image processing apparatus 100 has no frame buffer (which is used to store one full image) or block buffer (which is used to store an m×n block in an image, where the m×n block has m pixels in a width direction of the image and n pixels in a height direction of the image, and n is larger than 1) coupled between the output port P2 of the image pre-processing circuit 106 and the input port P3 of the SR circuit 102.
A frame buffer/block buffer is generally implemented by a dynamic random access memory (DRAM) or a static random access memory (SRAM). As a result, the frame buffer/block buffer occupies a part of the chip area and introduces image data transfer latency. Since the SR circuit 102 and/or the image pre-processing circuit 106 employ raster scan order processing and the image processing apparatus 100 does not need to use a frame buffer/block buffer between image pre-processing circuit 106 and SR circuit 102 and/or a frame buffer/block buffer between SR circuit 102 and resizer circuit 104, a low cost, low latency and low bandwidth processing apparatus can be achieved.
It should be noted that any image processing apparatus using a fixed-ratio SR circuit (e.g., an SR circuit with aid of AI) and an arbitrary-ratio resizer circuit (e.g., a resizer circuit without aid of AI) to achieve arbitrary-ratio scaling with image enhancement (e.g., arbitrary-ratio scaling with AI-aided image enhancement) falls within the scope of the present invention. For example, the image processing apparatus 100 shown in
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
This application claims the benefit of U.S. provisional application No. 62/987,962, filed on Mar. 11, 2020 and incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62987962 | Mar 2020 | US |
