Patent Application
20240119559
The present disclosure relates to an image enlarging apparatus and an image enlarging method thereof having deep learning mechanism.
Conventional image enlarging technologies are not able to increase the resolution of an enlarged image. As a result, blurriness, unclear edges and noises are easily observed in the enlarged image. In recent years, the technology of image super resolution is widely used in daily life, in which the object of such a technology is to obtain a high-resolution (HR) image from a low-resolution (LR) image and keeps the details thereof as much as possible.
Along with the improvement of the current digital resolution of the display apparatuses, the image enlarging technology equipped with the image super resolution mechanism is even more important while the resolution improves from full high definition (HD) to ultra HD. Whether the enlarged image can be adjusted as a whole and be enhanced according to local image characteristics at the same time becomes a critical issue.
In consideration of the problem of the prior art, an object of the present disclosure is to provide an image enlarging apparatus and an image enlarging method thereof having deep learning mechanism.
The present invention discloses an image enlarging apparatus having deep learning mechanism that includes a deep learning circuit, an image concatenating circuit and a super-resolution image enlarging circuit. The deep learning circuit includes an image downsizing circuit, an image characteristic analyzing circuit, a weighting reallocating circuit and an image upsizing circuit. The image downsizing circuit is configured to downsize an input image to generate a downsized image. The image characteristic analyzing circuit is configured to analyze the downsized image according to a plurality of image characteristics to generate a categorization map. The weighting reallocating circuit is configured to perform weighting reallocating on the categorization map according to a plurality of groups of image weighting parameters corresponding to the image characteristics to generate a weighting reallocating map. The image upsizing circuit is configured to upsize the weighting reallocating map to generate an adjusting map. The image concatenating circuit is configured to concatenate the input image and the adjusting map to generate a concatenated image. The super-resolution image enlarging circuit is configured to perform super-resolution enlarging on the concatenated image to generate an output image.
The present invention also discloses an image enlarging method having deep learning mechanism used in an image enlarging apparatus that includes steps outlined below. An input image is downsized to generate a downsized image by an image downsizing circuit included by a deep learning circuit. The downsized image is analyzed according to a plurality of image characteristics to generate a categorization map by an image characteristic analyzing circuit included by the deep learning circuit. Weighting reallocating is performed on the categorization map according to a plurality of groups of image weighting parameters corresponding to the image characteristics to generate a weighting reallocating map by a weighting reallocating circuit included by the deep learning circuit. The weighting reallocating map is upsized to generate an adjusting map by an image upsizing circuit included by the deep learning circuit. The input image and the adjusting map are concatenated to generate a concatenated image by an image concatenating circuit. Super-resolution enlarging is performed on the concatenated image to generate an output image by a super-resolution image enlarging circuit.
These and other objectives of the present disclosure will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiments that are illustrated in the various figures and drawings.
An aspect of the present invention is to provide an image enlarging apparatus and an image enlarging method thereof having deep learning mechanism to perform deep learning on the input image according to different image characteristics by a deep learning circuit disposed on a path independent from the image enlarging processing path so as to enhance the enlarged input image to accomplish an image enlarged result having super resolution mechanism applied thereto.
Reference is now made to
In an embodiment, the input image LR has an original size of H×W and an original channel number of Co, in which H is height, W is width and Co corresponds to such as, but not limited to different color channels. For example, the input image LR may have the original channel number of 3 (Co=3) corresponding to a red color channel (R), a green color channel (G) and a blue color channel (B). When the size ratio between the output image HR and the input image LR is n, the output image HR has an output size of nH×nW and an output channel number that equals to the original channel number Co.
The image enlarging apparatus 100 includes a deep learning circuit 110, an image concatenating circuit 120, a super-resolution image enlarging circuit 130 (abbreviated as SRI in
The deep learning circuit 110 includes the an image downsizing circuit 150, an image characteristic analyzing circuit 160, a weighting reallocating circuit 170 and an image upsizing circuit 180.
The image downsizing circuit 150 is configured to downsize the input image LR to generate a downsized image DR. In an embodiment, the image downsizing circuit 150 downsizes the input image LR with a ratio “s”. As a result, the downsized image DR has a downsized size of H/s×W/s and has the original channel number of Co.
The image characteristic analyzing circuit 160 is configured to analyze the downsized image LR according to a plurality of image characteristics to generate the categorization map AR. In an embodiment, the categorization map AR has the downsized size of H/s×W/s and has an analysis channel number Cs, wherein the analysis channel number Cs is the number of the image characteristics.
In an embodiment, the image characteristic analyzing circuit 160 is a semantic segmentation circuit, a local frequency detection circuit or a combination thereof. Further, the image characteristic analyzing circuit 160 may use the filtering technology of Sobel or discrete cosine transform (DCT) to perform analysis according to practical requirements.
The image characteristics include such as, but not limited to a plurality of object classes, a plurality of frequency ranges or a combination thereof. The object classes may include such as, but not limited to a plant, a building, a sky or other objects. The frequency ranges may include such as, but not limited to a high frequency range (e.g., a larger variation of parameters of the image) and a low frequency range (e.g., a smaller variation of parameters of the image).
In an embodiment, an object class or a frequency range corresponds to one analysis channel. For example, when the analysis channel number Cs is 3, such 3 analysis channels correspond to the plant, the building and the sky respectively. When the image characteristic analyzing circuit 160 determines that a pixel in the downsized image LR corresponds to the plant, the values of these 3 analysis channels are 1, 0 and 0, in which these values indicate that such a pixel corresponds to the plant instead of either the building or the sky.
The weighting reallocating circuit 170 is configured to perform weighting reallocating on the categorization map AR according to a plurality of groups of image weighting parameters corresponding to the image characteristics to generate a weighting reallocating map WR.
In an embodiment, the weighting reallocating map WR has the downsized size of H/s×W/s and a weighting reallocating channel number of Cw, in which the weighting reallocating channel number is the number of these groups of the image weighting parameters. In an embodiment, the groups of the image weighting parameters correspond to a de-noise process, a sharpness adjusting process, a texture enhancement processor a combination thereof.
In an embodiment, a group of the image weighting parameters corresponds to a weighting reallocating channel. For example, the weighting reallocating channel number is 2, and a group of parameters corresponding to the de-noise process and another group of parameters corresponding to the texture enhancement process are correspondingly generated for such 2 weighting reallocating channels according to the categorization map AR. More specifically, in an embodiment, when the categorization map AR includes a channel corresponding to the object class of sky, the weighting reallocating circuit 170 may generate the weighting reallocating map WR that includes a group of image weighting parameters corresponding to the de-noise process such that the sky in the image becomes clean after being processed by such a group of image weighting parameters.
The image upsizing circuit 180 is configured to upsize the weighting reallocating map WR to generate an adjusting map MR.
In an embodiment, the image upsizing circuit 180 upsizes the weighting reallocating map WR with a ratio “s”. As a result, the adjusting map MR has the original size of H×W and a weighting reallocating channel number of Cw.
The image concatenating circuit 120 is configured to concatenate the input image LR and adjusting map MR to generate a concatenated image CR.
In an embodiment, the concatenated image CR has the original size of H×W and a concatenated channel number Cc, wherein the concatenated channel number Cc is a sum of the original channel number Co and the weighting reallocating channel number Cw, namely Cc=Co+Cw.
The super-resolution image enlarging circuit 130 is configured to perform super-resolution enlarging on the concatenated image CR to generate the output image HR.
The concatenated image CR not only includes the input image LR, but also includes the additional adjusting map MR. As a result, the super-resolution image enlarging circuit 130 uses the adjusting map MR as control items to perform pixel-by-pixel modification to enhance the enlarged input image LR to accomplish an image enlarged result having super resolution mechanism applied thereto.
In an embodiment, a first frame refresh rate of the input image LR is N times of a second frame refresh rate of the adjusting map MR.
In an embodiment, the first frame refresh rate of the input image LR can be the same as the second frame refresh rate of the adjusting map MR (i.e., N=1), such that the speed that the deep learning circuit 110 generates the adjusting map MR is the same as the frame refresh rate of the input image LR. Under such a condition, the image concatenating circuit 120 may directly receive the adjusting map MR from the image upsizing circuit 180 to be concatenated with the input image LR.
However, since the processing of the deep learning circuit 110 is actually performed on a path independent from the image enlarging path, in another embodiment, the deep learning circuit 110 may be configured to set the second frame refresh rate of the adjusting map MR to be smaller than the first frame refresh rate of the input image LR. For example, when the first frame refresh rate of the input image LR is 60 Hz, the deep learning circuit 110 may set the second frame refresh rate of the adjusting map MR to be 30 Hz (i.e., N=2). As a result, the adjusting map MR has a frame delay relative to the input image LR.
Under such a condition, the memory circuit 140 is configured to store the adjusting map MR, such that the image concatenating circuit 120 is configured to retrieve the adjusting map MR through the dashed line path to be concatenated with the input image LR. Under the condition that the first frame refresh rate is two times of the second frame refresh rate described above, the image concatenating circuit 120 concatenates each of two consecutive input images LR with the same adjusting map MR so as to generate two concatenated images CR and perform super-resolution enlarging thereon to generate two output images HR.
Under such a condition, the system operation amount and the bandwidth of the image enlarging apparatus 100 can be adjusted dynamically. Since the adjusting map MR is not needed to be synchronous to the input image LR, the resource of the image enlarging apparatus 100 is not over-consumed.
It is appreciated that in practical implementation, the circuits included in the deep learning circuit 110 may store the processed content in the memory circuit 140 during the processing performed on the images and the maps so as to be retrieved by other circuits without being transmitted between the circuits. The present invention is not limited to the signal transmission method illustrated in
The image enlarging apparatus having deep learning mechanism of the present invention performs deep learning on the input image according to different image characteristics by a deep learning circuit disposed on a path independent from the image enlarging processing path so as to enhance the enlarged input image to accomplish an image enlarged result having super resolution mechanism applied thereto.
It is appreciated that the components included in the image enlarging apparatus 100 may be implemented by independent hardware circuits or by software modules generated by operation of software. The present invention is not limited thereto.
Reference is now made to
Besides the apparatus described above, the present invention further discloses the image enlarging method 200 that can be used in such as, but not limited to the image enlarging apparatus 100 illustrated in
In step S210, the input image LR is downsized to generate the downsized image DR by the image downsizing circuit 150 included by the deep learning circuit 110.
In step S220, the downsized image DR is analyzed according to the plurality of image characteristics to generate the categorization map AR by the image characteristic analyzing circuit 160 included by the deep learning circuit 110.
In step S230, weighting reallocating is performed on the categorization map AR according to the plurality of groups of image weighting parameters corresponding to the image characteristics to generate the weighting reallocating map WR by the weighting reallocating circuit 170 included by the deep learning circuit 110.
In step S240, the weighting reallocating map WR is upsized to generate the adjusting map MR by the image upsizing circuit 180 included by the deep learning circuit 110.
In step S250, the input image LR and the adjusting map MR are concatenated to generate the concatenated image CR by the image concatenating circuit 120.
In step S260, super-resolution enlarging is performed on the concatenated image CR to generate the output image HR by the super-resolution image enlarging circuit 130.
It is appreciated that the embodiments described above are merely an example. In other embodiments, it is appreciated that many modifications and changes may be made by those of ordinary skill in the art without departing, from the spirit of the invention.
In summary, the image enlarging apparatus and the image enlarging method thereof having deep learning mechanism perform deep learning on the input image according to different image characteristics by a deep learning circuit disposed on a path independent from the image enlarging processing path so as to enhance the enlarged input image to accomplish an image enlarged result having super resolution mechanism applied thereto.
The aforementioned descriptions represent merely the preferred embodiments of the present disclosure, without any intention to limit the scope of the present disclosure thereto. Various equivalent changes, alterations, or modifications based on the claims of present disclosure are all consequently viewed as being embraced by the scope of the present disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 111138225 | Oct 2022 | TW | national |
