Patent Application
20230130835
The present invention relates to image processing, and more particularly to an image processing system and a related image processing method for performing image enhancement based on region control and texture synthesis techniques.
Mid-frequency and high-frequency details of compressed videos or streaming videos are often lost due to compression algorithms. Typically, image enhancement can be relied upon to restore certain lost details. Common image enhancement approaches includes sharpening and deep-learning image enhancement. Sharpening generally involves increasing the high-frequency details in image, such as using a high-pass filter to enhance textures and edge areas in the images. However, sharpening cannot restore the textures and edges that have been completely destroyed during compression. On the other hand, deep learning image enhancement trains an image enhancement model by inputting a large number of various images to the model, allow the model to learn relationship between image contents, details and textures. When enhancing compressed images, the trained image enhancement model can guess what kind of details and textures are lost based on the image contents, and accordingly regenerate it. However, the disadvantage of deep learning image enhancement is that the regenerated texture and details are difficult to control, which may lead to unnatural artifacts. Also, deep-learning image enhancement requires higher computing power.
In view of this, the present invention provides an image enhancement processing technique based on texture synthesis and region control. The image enhancement processing of the present invention has the ability to generate details, and can provide a decent enhancement effect even when details of source images are completely lost. Since the present invention does not utilize a deep learning network to generate details, the computing power is not critical. In various embodiments of the present invention, a material image generating circuit is utilized to generate material images, and texture characteristics of the material images are adjusted through one or more texture generating circuit, thereby to generate texture images. Textures in the texture images may have specific directionalities and densities. In embodiments of the present invention, multiple texture images are generated by different configurations, thereby improving the adaptability to restoring different types of details in source images. After that, based on the analysis of regional characteristics of the source image (such as frequency, brightness, semantic segmentation or motion of object), synthesis intensities of texture images are regionally controlled to improve adjustability and matching degree between generated textures and lost details of the source images. As such, the image enhancement effects of texture images can be regionally controller, thereby achieving better and more natural results.
According to one embodiment, an image processing system is provided. The image processing system comprises: a material image generating circuit, at least one texture generating circuit and an output controller. The material image generating circuit is configured to generate a material image. The at least one texture generating circuit is coupled to the material image generating circuit, and configured to adjust texture characteristics of the material image to generate at least one texture image. The output controller is coupled to the at least one texture generating circuit, and configured to analyze regional characteristics of a source image to generate an analysis result, determine a region weight according to the analysis result, and synthesize the source image with the at least one texture image according to the region weight, thereby to generate an output image.
According to one embodiment, an image processing method is provided. The image processing method comprises: generating a material image, adjusting texture characteristics of the material image to generate at least one texture image; analyzing regional characteristics of a source image to generate an analysis result; determining a region weight according to the analysis result; and synthesizing the source image with the at least one texture image according to the region weight, thereby to generate an output image.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present embodiments. It will be apparent, however, to one having ordinary skill in the art that the specific detail need not be employed to practice the present embodiments. In other instances, well-known structures, materials or steps have not been presented or described in detail in order to avoid obscuring the present embodiments.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present embodiments. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable combinations and/or sub-combinations in one or more embodiments.
Please refer to
The function of the material image generating circuit 110 is to generate a material image IMG_MA (whose image size is H×W, identical to the size of the source image IMG_S). In one embodiment, the material image generating circuit 110 may be a random noise generating circuit, which may generate an image having noise with a random distribution. In some embodiments, the random noise generating circuit can be implemented by a linear feedback shift register (LFSR) or a hardware random number generator (HRNG) using thermal noise. The material image IMG_MA generated by the material image generating circuit 110 will be provided to the texture generating circuits 120_1 and 120_2.
The function of the texture generating circuits 120_1-120_2 is to adjust texture characteristics of the material image IMG_MA, and convert the material image IMG_MA into textures with a specific preference and distribution. In one embodiment, each of the texture generating circuits 120_1-120_2 includes one or more filters, which adjust the directionality and the density of the noise in the material image IMG_MA. In the embodiment shown in
The output controller 130 includes a region analysis circuit 132, a weight generating circuit 134, multiplying units 136_1-136_2, and adding units 138_1-138_2. The function of the output controller 130 is to detect regional characteristics of the source image IMG_S, and perform weight control on texture synthesis accordingly, thereby to achieve a decent image enhancement effect. The region analysis circuit 132 is operable to perform region analysis on the source image IMG_S. As shown in
The weight generating circuit 134 generates a region weight A corresponding to the texture image IMG_TX1 and a region weight B corresponding to the texture image IMG_TX2 according to the analysis result. As shown
Furthermore, directionality and density of textures in the texture image IMG_TXT1 and the texture image IMG_TXT2 may lead to their respective suitability for enhancing of different types of image contents and details. For example, the texture image IMG_TXT1 may be relatively suitable for enhancing details in darker regions, while the texture image IMG_TXT2 may be relatively suitable for enhancing details in brighter regions. In one embodiment, the texture image IMG_TXT1 may be relatively suitable for enhancing details of the grass, while the texture image IMG_TXT2 may be relatively suitable for enhancing details of the water surface. In one embodiment, the texture image IMG_TXT1 may be relatively suitable for enhancing details of objects in motion, while the texture image IMG_TXT2 may be relatively suitable for enhancing details of motionless objects. After obtaining the analysis result generated by the region analysis circuit 132 regarding the regional characteristics of the source image IMG_S, the weight generating circuit 134 can determine the region weights A and B according to the texture characteristics of the texture images IMG_TXT1 and IMG_TXT2, thereby accentuating or reducing the influence of texture images IMG_TXT1 and IMG_TXT2 on a specific region of the source image IMG_S. For example, if the texture characteristics of a texture image are suitable for a specific region of the source image, the weight corresponding to the specific region is accentuated (i.e., adaptive detail enhancement). On the other hand, if the texture characteristics of a texture image are not suitable for a specific region of the source image, the weight corresponding to the specific region is reduced (i.e., adaptive detail reduction). It is also available to accentuate the weights of all the texture images with respect to the specific region or reduce the weights of all the texture images with respect to the specific region. Once the weight generating circuit 134 determines the region weights A and B, the multiplying units 136_1-136_2 and the adding units 138_1-138_2 can use the weight coefficients A0-A23 and B0-B23 to synthesize the source image IMG_S with the texture images IMG_TXT1 and IMG_TXT2, thereby to produce the output image IMG_OUT.
It can be understood from the above descriptions that the texture image IMG_TXT1 and the texture image IMG_TXT2 will affect the adaptability of the image processing system 100 to processing source images with different contents. Therefore, in other embodiments of the present invention, the image processing system 100 may have more texture generating circuits to generate more texture images having different directionalities and different densities in texture distribution, so as to restore details for images with various details better. In addition, in embodiments shown in
In other embodiments of the present invention, the material image generating circuit can be implemented by a pattern extracting circuit. As shown in an embodiment of
S310: generating a material image;
S320: adjusting texture characteristics of the material image to generate at least one texture image;
S330: analyzing regional characteristics of a source image to generate an analysis result;
S340: determining a region weight according to the analysis result; and
S350: synthesizing the source image with the at least one texture image according to the region weight, thereby to generate an output image.
Since the principle and specific details of the foregoing steps have been described expressly in the above embodiments, further description will not be repeated here. It should be noted, that the above flow may be able to achieve better enhancement processing and further improve enhancement effect by adding other extra steps or making appropriate modifications and adjustments. Furthermore, all the operations in the above embodiments of the present invention can be implemented by a device 400 shown in
In summary, the image enhancement processing of the present invention has the ability to produce details, so it can still exert a certain enhancement effect even when details of the source image are completely lost. As the deep learning network is not utilized to generate details in the present invention, the requirements on computing resources are relatively low. In embodiments of the present invention, the material image generating circuit or the pattern extracting circuit is utilized to generate the material image, and one or more texture generating circuits are utilized to adjust the texture characteristics of the material image to generate texture images. In the embodiments of the present invention, multiple texture images are generated by different settings, thereby improving the adaptability to restoring different types of lost details in the source image. After that, the regional characteristics of the source image (such as, frequency, brightness, semantic segmentation or object motion) are analyzed. When the source image are synthesized with the texture images, the intensity of the enhancement effect is able to be controlled by regions, thereby improving adjustability, and also improving the matching degree of generated texture and lost details in the source image, achieving better and more natural image enhancement effects.
Embodiments in accordance with the present embodiments can be implemented as an apparatus, method, or computer program product. Accordingly, the present embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects that can all generally be referred to herein as a “module” or “system.” Furthermore, the present embodiments may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium. In terms of hardware, the present invention can be accomplished by applying any of the following technologies or related combinations: an individual operation logic with logic gates capable of performing logic functions according to data signals, and an application specific integrated circuit (ASIC), a programmable gate array (PGA) or a field programmable gate array (FPGA) with a suitable combinational logic.
The flowchart and block diagrams in the flow diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. These computer program instructions can be stored in a computer-readable medium that directs a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 110139490 | Oct 2021 | TW | national |
