Patent Application
20180025474
The disclosure is based on and claims the benefits of priority to Chinese Application No. 201610574955.7, filed Jul. 20, 2016, the entire contents of which are incorporated herein by reference.
The disclosure relates to a field of computers, and in particular, to video processing technology.
In video encoding processing, noise processing has always been a very challenging aspect. There are many sources for video noise, such as thermal noise during video capturing, encoding noise during video encoding, and processing noise in some video enhancement processing. The video noise may deteriorate viewing experience of users viewing the video images, increase the difficulty of video compression, reduce a compression ratio, and affect accuracy of a video detection algorithm. Therefore, it is necessary to encode and denoise videos having significant noise. There are many denoising algorithms, such as a bilateral filter, a non-local mean filter, a guided filter, a block-matching and 3D (BM3D) filter, a block-matching and 4D (BM4D) filter and the like, and they have all been widely used in respective application fields.
But noise estimation methods have a common problem: the accuracy of their estimation results rely directly on the image content per se, and they cannot distinguish whether the extracted high frequency energy is from a signal or from noise. That is, a precondition that they can obtain a more accurate result is that the high frequency energy of the signal is only distributed in a small proportion of image regions. In other words, it is expected that, the video has more uniform blocks or static blocks, which cannot be met in many cases.
Another common problem of the denoising algorithms is the huge cost in processing time. Taking the BM4D filter as an example, it takes tens of seconds for processing one frame on a computer based on an Intel® i7 quad-core CPU. Therefore, the BM4D filter is highly unlikely to be used in a video encoder. On the other hand, the guided filter has the lowest complexity, but one main problem of the guided filter is that those regions with many details cannot be processed, because the guided filter cannot calculate matching blocks, and therefore having a weak noise suppression effect on regions having great high frequency energy for signals.
An objective of the present application is to provide a video processing method and device, which can solve the problem that the existing solution cannot accurately denoise videos.
Some embodiments of the disclosure provide a video processing method, comprising: estimating noise of an input video to obtain noise level information of the input video; and selecting, according to the noise level information, a denoising scheme to process the input video.
Some embodiments of the disclosure provide a video processing apparatus, comprising a noise estimation device used for estimating noise of an input video to obtain noise level information of the input video; and a processing device used for selecting, according to the noise level information, a denoising scheme to process the input video.
The drawings herein are provided, as a part of the disclosure, for further understanding of the disclosure. Illustrative embodiments of the disclosure and description thereof are used to explain the disclosure, and are not restrictive. In the drawings,
Reference will now be made in detail to the disclosed embodiments, examples of which are illustrated in the accompanying drawings. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
In the video processing method according to some embodiments of the disclosure, noise of an input video is estimated to obtain noise level information, and a corresponding denoising scheme is selected based on the noise level information to process the input video. Further, in the denoising scheme, multiple denoising algorithms can be merged to achieve an appropriate denoising effect.
In some embodiments, a video processing system may be provided.
These modules in the video processing system cooperate with each other to accomplish denoising work of an input video. For example, after the denoising input module 201 acquires an input video, the denoising input module 201 can provide the input video to the noise estimation module 203 and the noise elimination module 205. The noise estimation module 203 performs noise estimation on the input video to obtain noise level information. The noise elimination module 205 selects a denoising scheme according to the noise level information from noise estimation module 203 and system complexity control from complexity control module 207, to perform a reasonable and efficient noise elimination process on the input video. While it is preferable that the noise elimination module 205 eliminates any noise, it is appreciated that the noise elimination module 205 can minimize any noise. After reducing or elimination the noise on the input video, noise elimination module outputs the denoised video to denoising output module 209.
The video processing system supports customization according to system complexity. And the video processing system may further flexibly adjust the denoising scheme and a denoising algorithm in the denoising scheme according to a relationship between actual complexity and preset values during the processing. Thus, the best denoising requirement may be met under a limited complexity, and a compression ratio and compression quality of the input video may be improved.
Steps S11 and S12 as shown in
In step S111, original image data and corresponding filtered image data of a frame of the plurality of frames from the input video are acquired.
In step S112, noise estimation information of the frame is determined based on statistics information of the original image data and the corresponding filtered image data of the frame.
In step S113, smoothed noise estimation information is obtained according to the noise estimation information of the frame, and the smoothed noise estimation information is used as the noise level information of the input video.
If it can be distinguished during the noise estimation that whether the extracted high frequency energy is a signal or noise, the accuracy of the noise estimation will be improved. In some embodiments, high frequency components of video content can contain more energy after a low-pass filtering. In some embodiments, original image data and corresponding filtered image data of a frame of an input video are obtained through an input video. A mathematical calculation is performed on the original image data and the corresponding filtered image data of the frame. For example, a block variance calculation can be performed to obtain corresponding statistics information. And noise estimation information of the frame can be determined by comparing statistics information of the original image data and the corresponding filtered image data of the frame. Since the result of single-frame noise estimation has a large jitter near a true value, it is inaccurate if single-frame noise estimation information is used as noise estimation of the input video. Thus, an error of noise estimation can be reduced by multi-frame joint estimation, and the result of a multi-frame noise estimation is given by merging single-frame noise levels. For example, a relatively smooth estimation value is obtained by effectively estimating an average of single-frame noise levels within a time window, significantly improving accuracy and robustness of the noise estimation.
In some embodiments, the statistics information may include original statistics information and processing statistics information. Therefore, step S112 may include: performing a block variance calculation on the original image data and the corresponding filtered image data of the frame respectively, to obtain corresponding original statistics information and processing statistics information; and comparing the original statistics information of the frame with the processing statistics information, to determine the noise estimation information of the frame.
An algorithm for noise estimation will be further described as below.
In some embodiments, in step S12 of
As shown in
When determining whether the input video is a video-to-be-denoised, the noise level information may be compared with a preset denoising threshold. For example, an estimation threshold may be set according to the noise level information. Statistics is made on a total number of frames having single-frame noise level greater than the estimation threshold in the noise estimation information. Next, whether the total number of frames is greater than a preset denoising threshold (a number threshold) is determined. And if the total number of frames is greater than a preset denoising threshold, the input video is determined as the video-to-be-denoised. When the input video is not a video-to-be-denoised, the denoising scheme directly outputs the input video. And when the input video is determined as the video-to-be-denoised, the denoising scheme maximizes a denoising effect according to an automatic complexity limitation. The auto complexity limitation may limit the complexity of a denoising algorithm according to the noise level information of the video-to-be-denoised. That is, during a denoising process, the denoising algorithm with corresponding complexity may be selected according to the noise level information, so as to denoise the video-to-be-denoised.
It is appreciated that the comparing of the total number of frames with a preset denoising threshold is merely an illustrated method of determining whether an input video is a video-to-be-denoised. And other methods of determining an input video as a video-to-be-denoised may be applicable to embodiments of the disclosure and should also be included within the scope of the disclosure.
In some embodiments, selecting a corresponding denoising algorithm of the video-to-be-denoised according to the noise level information in step S12 includes: determining a denoising parameter of the video-to-be-denoised according to the noise level information; and determining the denoising algorithm of the video-to-be-denoised according to the denoising parameter.
When the video-to-be-denoised is denoised, a reasonable denoising parameter is determined according to noise level information of the video-to-be-denoised at first. Then, a denoising scheme merged from one or more denoising algorithms is determined according to the denoising parameter.
In some embodiments, determining a denoising parameter of the video-to-be-denoised according to the noise level information includes: determining the denoising parameter of the video-to-be-denoised according to the noise level information and encoder information, as further discussed with reference to
The system according to some embodiments of the disclosure provides a denoising module cooperating with an encoder, so as to improve the processing effect of the encoder on the input video. The encoder in the system can be an encoder having a video denoising function. The system provides some schemes of denoising in cooperation with encoding processing, and provides a scheme of a video denoising system with low complexity and high quality as well. According to an actual test result, for the input video, the encoder having a denoising function can evidently improve the video image quality while saving 30+ % of the code rate.
It should be noted that, the cooperation of the denoising module and the encoding module is an example according to embodiments of the disclosure. And cooperation of other methods on video denoising and other applications may be applicable to embodiments of the disclosure, and should also be included within the scope of the disclosure.
In some embodiments, the denoising algorithm of the video-to-be-denoised includes: denoising the video-to-be-denoised by an inter-frame filtering algorithm.
Time domain noise in the video-to-be-denoised seriously affects the feeling of viewing and the compression ratio. For example, if the noise level information is relatively small (i.e., there is a little noise in the video-to-be-denoised), an inter-frame filtering algorithm can be selected to denoise the video-to-be-denoised, according to a real-time demand of an application scenario. Removal of the time domain noise ensures the image quality and the compression ratio of the processed video. In some embodiments, denoising the video-to-be-denoised by an inter-frame filtering algorithm may include: calculating a motion index between a current frame of the video-to-be-denoised and an adjacent frame thereof, and denoising the video-to-be-denoised by the inter-frame filtering algorithm according to the motion index. In some embodiments, a motion-adaptive inter-frame denoising manner is adopted, so as to ensure the processing speed. That is, a motion index is obtained by calculating a pixel difference in corresponding positions of adjacent frames, and an inter-frame filtering weight is determined from the motion index. The smaller the motion index is, the greater the possibility that an object is stationary and a blurring image is therefore not easy to appear. And if the pixel difference in corresponding positions of adjacent frames is more than a certain threshold, points in the positions are not filtered. During the calculation of the motion index, the motion index can be determined according to differential values of pixels. For example, a current pixel in a position (i,j) of a current frame Xn is Yn(i,j), the motion index is denoted as Mn(i,j), and then Mn(i,j)=Yn(i,j)−Yn-1(i,j). It should be noted that, the above differential calculation may be used as a method of comparing pixel differences, and other mathematical comparison calculation methods or determinations may also be used. A filtering result from the inter-frame filtering can be obtained according to the motion index and the determined inter-frame filtering weight. And there is a mapping curve relationship between the inter-frame filtering weight and the motion index.
In some embodiments, a set of video images has 5 frames. A current frame to be filtered is the third frame. An inter-frame filtering result is obtained by performing a weighted calculation according to inter-frame filtering weights of the second frame, the third frame and the fourth frame. The inter-frame filtering weights of the second frame, the third frame and the fourth frame are denoted as S2, S3, and S4 respectively, and the corresponding pixel values are denoted as A2, A3, and A4. Therefore, the output inter-frame filtering result is (S2×A2+S3×A3+S4×A4).
It should be noted that, the previous few frames and the next few frames adjacent to the current frame may also be selected. For example, when the third frame is to be filtered, parameters of the previous two frames (the first frame and the second frame) and the next two frames (the fourth frame and the fifth frame) may be selected to calculate an inter-frame filtering result, and it is not merely limited to only using parameters of one adjacent frame both before and after the current frame to calculate the inter-frame filtering result.
After the inter-frame filtering is performed to denoise the time domain noise, the denoised filtering result is output.
In some embodiments, after denoising the video-to-be-denoised by an inter-frame filtering algorithm, the method further includes: denoising the video-to-be-denoised by a simple intra-frame filtering algorithm; and merging denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm and outputting the merged result.
According to a real-time demand of an application scenario, when the video-to-be-denoised is denoised by an inter-frame filtering algorithm, one parameter may be set to evaluate whether the inter-frame filtering meets the requirement. For example, it is required that denoising with how many points is accomplished within how much time, or an inter-frame filtering algorithm is used to denoise the previous few frames, while there is a higher requirement for the image quality or the compression ratio of the next few frames according to system demands. A denoising scheme is selected based on a denoising parameter set according to the noise level information. In this case, the denoising scheme performs denoising by a simple intra-frame filtering algorithm and an inter-frame filtering algorithm, as a complement to each other.
The simple intra-frame filtering algorithm includes filtering algorithms having low complexity and high calculation efficiency. That is, the filtering algorithms do not necessarily need complicated matching logic and may only need some simple calculation of statistical values. For example, a mean filter and a guided filter both belong to the simple intra-frame filtering algorithm and may complement the inter-frame filtering algorithm. The final denoising result is obtained after denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm are merged, and the denoising result is output.
In some embodiments, denoising the video-to-be-denoised by a simple intra-frame filtering algorithm includes: calculating a smoothing index in the video-to-be-denoised, and denoising the video-to-be-denoised by the simple intra-frame filtering algorithm according to the smoothing index.
In some embodiments, when the simple intra-frame filtering algorithm and the inter-frame filtering algorithm are merged for use, a guided filter may be employed as the simple intra-frame filtering algorithm. The guided filter has high calculation efficiency, and has an excellent processing effect on a smooth region. A smoothing index in the video-to-be-denoised may be calculated and the result of the simple intra-frame filtering algorithm may be determined according to the smoothing index.
The inter-frame filtering algorithm has a good denoising effect for a stationary low-changing region, the simple intra-frame filtering algorithm has a good effect for the smooth region, and they can cooperate with each other to cover most of the video regions. Therefore, upon completion of the processing via the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, a good denoising effect can be ensured for stationary and smooth regions.
In some embodiments, merging denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm and outputting the merged result includes: determining denoising strength information based on the smoothing index and the motion index; and determining whether the denoising strength information is greater than a screening threshold, and if the denoising strength information is greater than or equal to a screening threshold, merging denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm and outputting the merged result.
In some embodiments, after the video-to-be-denoised has been denoised by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, denoising strength information is determined according to the smoothing index and the motion index. For example, in the weighted filtering, a denoising result is obtained by weighted merging of an inter-frame filtering result and a simple intra-frame filtering result. The greater the weight is, the stronger the filtering effect is and the greater the denoising strength is.
After the denoising strength information is obtained, whether the denoising strength information is greater than a screening threshold is determined. And if the denoising strength information is greater than or equal to a screening threshold, the denoising scheme meets the video processing demand, and the inter-frame filtering result and the simple intra-frame filtering result are merged and then the merged result is output.
In some embodiments, after denoising the video-to-be-denoised by an inter-frame filtering algorithm, the method further includes: denoising the video-to-be-denoised by a simple intra-frame filtering algorithm; denoising the video-to-be-denoised by a complicated intra-frame filtering algorithm; and merging denoising results of the inter-frame filtering algorithm, the simple intra-frame filtering algorithm and the complicated intra-frame filtering algorithm and outputting the merged result.
As for videos having obvious noise, the noise is suppressed as much as possible on the premise of being allowed by complexity. The denoising scheme selected according to a set denoising parameter and a complexity control demand may further include: denoising the video-to-be-denoised by the complicated intra-frame filtering algorithm after the video-to-be-denoised is denoised by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm.
And denoising the video-to-be-denoised by the simple intra-frame filtering algorithm includes: calculating a smoothing index in the video-to-be-denoised, and denoising the video-to-be-denoised by the simple intra-frame filtering algorithm according to the smoothing index.
The complicated intra-frame filtering algorithm refers to a filter having high complexity and requiring a great amount of match-related calculations. For example, the complicated intra-frame filtering algorithm may include a bilateral filter, a non-local mean filter, a BM3D filter, and a BM4D filter. After videos are processed by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, most of the noise remains in moving detail regions.
In this case, a part of image regions can be selected for more complicated denoising according to different demands for complexity.
For example, filtering algorithms having high complexity, such as a BM3D filter, may be selected. Therefore, matching blocks may be addressed during the processing. These matching blocks are used for filtering. A good SNR result can be obtained by strong correlation of signals and non-correlation of noise. Thus, a better denoising effect may be achieved. Finally, denoising results of the denoising algorithms are merged and then the merged result is outputted.
In some embodiments, denoising the video-to-be-denoised by a complicated intra-frame filtering algorithm includes: determining denoising strength information of each point within a current frame of the video-to-be-denoised based on the smoothing index and the motion index; and determining whether the denoising strength information is greater than a screening threshold, and if the denoising strength information is not greater than a screening threshold, denoising the current frame by the complicated intra-frame filtering algorithm.
In some embodiments, after the video-to-be-denoised is denoised by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, denoising strength information of each point is calculated according to the smoothing index and the motion index. After the denoising strength information is obtained, whether the denoising strength information is greater than a screening threshold is determined. If the denoising strength information is not greater than a screening threshold, complicated intra-frame filtering can also be required.
In some embodiments, after the determining whether the denoising strength information is greater than a screening threshold, the method further includes: making statistics on point number information of the current frame after the complicated intra-frame filtering; and determining the screening threshold based on the point number information and a system complexity parameter. In this embodiment, statistics are made on the point number information of the frame on which complicated intra-frame filtering is performed, and then the screening threshold is adjusted according to the system complexity requirement. If the calculation complexity of the frame is greater than the requirement, then the screening threshold is reduced. The screening threshold is adjusted adaptively according to real-time denoising statistical information and a system complexity parameter, to meet the denoising effect with a given complexity limitation.
In some embodiments, three denoising algorithms may be used, and the three denoising algorithms are merged according to a certain strategy.
Steps as shown in
As shown in
As discussed above, denoising according to some embodiments of the disclosure may achieve a high-quality denoising effect with low complexity. The denoising may merge some classical denoising algorithms, flexibly select and combine different denoising algorithms according to system complexity configuration, and achieve the denoising requirement with limited complexity.
Noise estimation device 11 may estimate noise of an input video to obtain noise level information of the input video. Processing device 12 may select a denoising scheme corresponding to the noise level information process the input video.
Apparatus 1 may include but is not limited to any mobile electronic product that may interact with a user via a touch screen, such as a smart phone, a PDA, and the like. The mobile electronic product may employ any operating system, such as an Android operating system, an iOS operating system, and the like. In some embodiments, apparatus 1 may include a script program running on a user apparatus or an apparatus consisting of the user apparatus and a network device or a touch end device connected via the network. It is appreciated that, apparatus 1 described above is merely an example, and any apparatus functions similarly falls within the scope of the disclosure.
The above-described devices work continuously with each other. It is appreciated that, “continuously” indicates each of the devices works in real time or according to a work pattern that is preset or adjusted in real time.
The apparatus for video processing provided by some embodiments of the disclosure processes input video based on a denoising scheme corresponding to noise level information generated by performing a noise estimation on the input video. The denoising scheme may merge many denoising algorithms to provide an excellent denoising effect.
In some embodiments, a video processing system may be provided in apparatus 1.
These modules in the video processing system cooperate with each other to accomplish denoising work of an input video. For example, after the denoising input module 201 acquires an input video, the noise estimation module 203 performs noise estimation on the input video to obtain noise level information, and the noise elimination module 205 selects a denoising scheme according to the noise level information and system complexity control, to perform a reasonable and efficient noise elimination process on the input video, and outputs the denoised video. The video processing system supports customization of complexity, and flexibly adjusts the denoising scheme and a denoising algorithm in the denoising scheme according to a relationship between actual complexity and preset values during the processing, therefore meeting the best denoising requirement under a limited complexity and improving a compression ratio and compression quality of the input video. These modules (and any corresponding sub-modules or sub-units) can be packaged functional hardware units (e.g., portions of an integrated circuit) designed for use with other components or a part of a program (stored on a computer readable medium) that performs a particular function, corresponding to the particular step, of related functions.
Noise estimation device 11 may include an acquiring unit 111, a determining unit 112, and an updating unit 113.
Acquiring unit 111 may acquire original image data and corresponding filtered image data of a frame of a plurality of frames from the input video.
Determining unit 112 may determine noise estimation information of the frame based on statistics information of the original image data and the corresponding filtered image data of the frame.
Updating unit 113 may obtain, according to the noise estimation information of the frame, smoothed noise estimation information as the noise level information of the input video.
If it can be distinguished during the noise estimation that whether the extracted high frequency energy is a signal or noise, the accuracy of the noise estimation will be improved. In some embodiments, high frequency components of video content can contain more energy after a low-pass filtering. In some embodiments, original image data and corresponding filtered image data of a frame of an input video are obtained through an input video. A mathematical calculation is performed on the original image data and the corresponding filtered image data of the frame. For example, a block variance calculation can be performed to obtain corresponding statistics information. And noise estimation information of the frame can be determined by comparing statistics information of the original image data and the corresponding filtered image data of the frame. Since the result of single-frame noise estimation has a large jitter near a true value, it is inaccurate if single-frame noise estimation information is used as noise estimation of the input video. Thus, an error of noise estimation can be reduced by multi-frame joint estimation, and the result of a multi-frame noise estimation is given by merging single-frame noise levels. For example, a relatively smooth estimation value is obtained by effectively estimating an average of single-frame noise levels within a time window, significantly improving accuracy and robustness of the noise estimation.
In some embodiments, determining unit 112 may be configured to perform a block variance calculation on the original image data and the corresponding filtered image data of the frame respectively, to obtain corresponding original statistics information and processing statistics information; and comparing the original statistics information of the frame with the processing statistics information, to determine the noise estimation information of the frame.
For example, determining unit 112 may determine noise estimation information according to the exemplary noise estimation algorithm of
In some embodiments, processing device 12 may be configured to select a denoising scheme corresponding to the noise level information to process the input video by: comparing the noise level information with a preset denoising threshold, to determine whether the input video is a video-to-be-denoised; if the input video is not a video-to-be-denoised, the denoising scheme directly outputting the input video, and if the input video is a video-to-be-denoised, the denoising scheme selecting a corresponding denoising algorithm of the video-to-be-denoised according to the noise level information.
For example, processing device 12 may select a denoising scheme corresponding to the noise level information to process the input video according to the exemplary processing algorithm of
As shown in
When determining whether the input video is a video-to-be-denoised, the noise level information may be compared with a preset denoising threshold. For example, an estimation threshold may be set according to the noise level information. Statistics is made on a total number of frames having single-frame noise level greater than the estimation threshold in the noise estimation information. Next, whether the total number of frames is greater than a preset denoising threshold (a number threshold) is determined. And if the total number of frames is greater than a preset denoising threshold, the input video is determined as the video-to-be-denoised. When the input video is not a video-to-be-denoised, the denoising scheme directly outputs the input video. And when the input video is determined as the video-to-be-denoised, the denoising scheme maximizes a denoising effect according to automatic complexity limitations. During a denoising process, denoising algorithms with different complexity are selected according to the noise level information to denoise the video-to-be-denoised.
It is appreciated that, the comparing the total number of frames with a preset denoising threshold is merely an illustrated method of determining whether an input video is a video-to-be-denoised. And other methods of determining an input video as a video-to-be-denoised may be applicable to embodiments of the disclosure and should also be included within the scope of the disclosure.
In some embodiments, processing device 12 may be further configured to: determine a denoising parameter of the video-to-be-denoised according to the noise level information; and determine the denoising algorithm of the video-to-be-denoised according to the denoising parameter.
When the video-to-be-denoised is denoised, a reasonable denoising parameter is determined according to noise level information of the video-to-be-denoised at first. Then, a denoising scheme merged from one or more denoising algorithms is determined according to the denoising parameter.
In some embodiments, processing device 12 may be configured to: determine the denoising parameter of the video-to-be-denoised according to the noise level information and encoder information, as further discussed with reference to
The system according to some embodiments of the disclosure provides a denoising module cooperating with an encoder, so as to improve the processing effect of the encoder on the input video. The encoder in the system is an encoder having a video denoising function. The system provides some schemes of denoising in cooperation with encoding processing, and provides a scheme of a video denoising system with low complexity and high quality as well. According to an actual test result, for the input video, the encoder having a denoising function can evidently improve the video image quality while saving 30+ % of the code rate.
It should be noted that, the cooperation of the denoising module and the encoding module is an example according to embodiments of the disclosure. And cooperation of other methods on video denoising and other application may be applicable to embodiments of the disclosure, and should also be included within the scope of the disclosure.
In some embodiments, processing device 12 may be configured to denoise the video-to-be-denoised by an inter-frame filtering algorithm.
Time domain noise in the video-to-be-denoised seriously affects the feeling of viewing and the compression ratio. For example, if the noise level information is relatively small (i.e., there is a little noise in the video-to-be-denoised), an inter-frame filtering algorithm can be selected to denoise the video-to-be-denoised, according to a real-time demand of an application scenario. Removal of the time domain noise ensures the image quality and the compression ratio of the processed video.
In some embodiments, processing device 12 may be configured to calculate a motion index between a current frame of the video-to-be-denoised and an adjacent frame thereof, and denoise the video-to-be-denoised by the inter-frame filtering algorithm according to the motion index.
In some embodiments, a motion-adaptive inter-frame denoising manner is adopted, so as to ensure the processing speed. That is, a motion index is obtained by calculating a pixel difference in corresponding positions of adjacent frames, and an inter-frame filtering weight is determined from the motion index. The smaller the motion index is, the greater the possibility that an object is stationary and a blurring image is therefore not easy to appear. And if the pixel difference in corresponding positions of adjacent frames is more than a certain threshold, points in the positions are not filtered. During the calculation of the motion index, the motion index can be determined according to differential values of pixels. For example, a current pixel in a position (i,j) of a current frame Xn is Yn(i,j), the motion index is denoted as Mn(i,j), and then Mn(i,j)=Yn(i,j)−Yn-1(i,j). It should be noted that, the above differential calculation may be used as a method of comparing pixel differences, and other mathematical comparison calculation methods may also be used. A filtering result from the inter-frame filtering can be obtained according to the motion index and the determined inter-frame filtering weight. And there is a mapping curve relationship between the inter-frame filtering weight and the motion index.
In some embodiments, a set of video images has 5 frames. A current frame to be filtered is the third frame. An inter-frame filtering result is obtained by performing a weighted calculation according to inter-frame filtering weights of the second frame, the third frame and the fourth frame. The inter-frame filtering weights of the second frame, the third frame and the fourth frame are denoted as S2, S3 and S4 respectively, and the corresponding pixel values are denoted as A2, A3 and A4.
Therefore, the output inter-frame filtering result is (S2×A2+S3×A3+S4×A4).
It should be noted that, the previous few frames and the next few frames adjacent to the current frame may also be selected. For example, when the third frame is to be filtered, parameters of the previous two frames (the first frame and the second frame) and the next two frames (the fourth frame and the fifth frame) may be selected to calculate an inter-frame filtering result, and it is not merely limited to only using parameters of one adjacent frame both before and after the current frame to calculate the inter-frame filtering result.
After the inter-frame filtering is performed to denoise the time domain noise, the denoised filtering result is output.
In some embodiments, after denoising the video-to-be-denoised by an inter-frame filtering algorithm, processing device 12 may be configured to: denoise the video-to-be-denoised by a simple intra-frame filtering algorithm; and merge denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm and outputting the merged result.
According to a real-time demand of an application scenario, when the video-to-be-denoised is denoised by an inter-frame filtering algorithm, one parameter may be set to evaluate whether the inter-frame filtering meets the requirement. For example, it is required that denoising with how many points is accomplished within how much time, or an inter-frame filtering algorithm is used to denoise the previous few frames, while there is a higher requirement for the image quality or the compression ratio of the next few frames according to system demands. A denoising scheme is selected based on a denoising parameter set according to the noise level information. In this case, the denoising scheme performs denoising by a simple intra-frame filtering algorithm and an inter-frame filtering algorithm, as a complement to each other.
The simple intra-frame filtering algorithm includes filtering algorithms having low complexity and high calculation efficiency. That is, the filtering algorithms do not need complicated matching logic and only need some simple calculation of statistical values. For example, a mean filter and a guided filter both belong to the simple intra-frame filtering algorithm and may complement the inter-frame filtering algorithm. The final denoising result is obtained after denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm are merged, and the denoising result is output.
In some embodiments, processing device 12 may be configured to calculate a smoothing index in the video-to-be-denoised, and denoise the video-to-be-denoised by the simple intra-frame filtering algorithm according to the smoothing index.
In some embodiments, when the simple intra-frame filtering algorithm and the inter-frame filtering algorithm are merged for use, a guided filter may be employed as the simple intra-frame filtering algorithm. The guided filter has high calculation efficiency, and has an excellent processing effect on a smooth region. A smoothing index in the video-to-be-denoised may be calculated and the result of the simple intra-frame filtering algorithm may be determined according to the smoothing index.
The inter-frame filtering algorithm has a good denoising effect for a stationary low-changing region, the simple intra-frame filtering algorithm has a good effect for the smooth region, and they can cooperate with each other to cover most of the video regions. Therefore, upon completion of the processing via the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, a good denoising effect can be ensured for stationary and smooth regions.
In some embodiments, processing device 12 may be configured to determine denoising strength information based on the smoothing index and the motion index; and determine whether the denoising strength information is greater than a screening threshold, and if the denoising strength information is greater than or equal to a screening threshold, merge denoising results of the inter-frame filtering algorithm and the simple intra-frame filtering algorithm and output the merged result.
In some embodiments, after the video-to-be-denoised has been denoised by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, denoising strength information is determined according to the smoothing index and the motion index. For example, in the weighted filtering, a denoising result is obtained by weighted merging of an inter-frame filtering result and a simple intra-frame filtering result. The greater the weight is, the stronger the filtering effect is and the greater the denoising strength is.
After the denoising strength information is obtained, whether the denoising strength information is greater than a screening threshold is determined, and if the denoising strength information is greater than or equal to a screening threshold is determined, it indicates that the denoising scheme meets the video processing demand, and the inter-frame filtering result and the simple intra-frame filtering result are merged and then the merged result is output.
In some embodiments, after denoising the video-to-be-denoised by an inter-frame filtering algorithm, processing device 12 may be configured to denoise the video-to-be-denoised by a simple intra-frame filtering algorithm; denoise the video-to-be-denoised by a complicated intra-frame filtering algorithm; and merge denoising results of the inter-frame filtering algorithm, the simple intra-frame filtering algorithm and the complicated intra-frame filtering algorithm and outputting the merged result.
As for videos having obvious noise, the noise is suppressed as much as possible on the premise of being allowed by complexity. The denoising scheme selected according to a set denoising parameter and a complexity control demand may further include: denoising the video-to-be-denoised by the complicated intra-frame filtering algorithm after the video-to-be-denoised is denoised by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm.
And denoising the video-to-be-denoised by the simple intra-frame filtering algorithm includes: calculating a smoothing index in the video-to-be-denoised, and denoising the video-to-be-denoised by the simple intra-frame filtering algorithm according to the smoothing index.
The complicated intra-frame filtering algorithm refers to a filter having high complexity and requiring a great amount of match-related calculation. For example, the complicated intra-frame filtering algorithm may include a bilateral filter, a non-local mean filter, a BM3D filter, and a BM4D filter. After videos are processed by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, most of the noise remains in moving detail regions.
In this case, a part of image regions can be selected for more complicated denoising according to different demands for complexity.
For example, filtering algorithms having high complexity, such as a BM3D filter, are selected. Therefore, matching blocks may be addressed during the processing. These matching blocks are used for filtering. A good SNR result can be obtained by strong correlation of signals and non-correlation of noise. Thus, a better denoising effect may be achieved. Finally, denoising results of the denoising algorithms are merged and then the merged result is output.
In some embodiments, processing device 12 may be configured to determine denoising strength information of each point within a current frame of the video-to-be-denoised based on the smoothing index and the motion index; and determine whether the denoising strength information is greater than a screening threshold, and if the denoising strength information is not greater than a screening threshold, denoise the current frame by the complicated intra-frame filtering algorithm.
In some embodiments, after the video-to-be-denoised is denoised by the inter-frame filtering algorithm and the simple intra-frame filtering algorithm, denoising strength information of each point is calculated according to the smoothing index and the motion index. After the denoising strength information is obtained, whether the denoising strength information is greater than a screening threshold is determined. If the denoising strength information is not greater than a screening threshold, complicated intra-frame filtering is also required.
In some embodiments, processing device 12 may be further configured to make statistics on point number information of the current frame after the complicated intra-frame filtering; and determine the screening threshold based on the point number information and a system complexity parameter. In this embodiment, statistics is made on the point number information of the frame on which complicated intra-frame filtering is performed, and then the screening threshold is adjusted according to the system complexity requirement. If the calculation complexity of the frame is greater than the requirement, then the screening threshold is reduced. The screening threshold is adjusted adaptively according to real-time denoising statistical information and a system complexity parameter, to meet the denoising effect with a given complexity limitation.
For example, processing device 12 may process an original video or input video according to the exemplary denoising algorithm of
In some embodiments, three denoising algorithms may be used, and the three denoising algorithms are merged according to a certain strategy.
Steps as shown in
As shown in
As discussed above, denoising according to some embodiments of the disclosure may achieve a high-quality denoising effect with low complexity. The denoising may merge some classical denoising algorithms, flexibly select and combine different denoising algorithms according to system complexity configuration, and achieve the denoising requirement with limited complexity.
It is appreciated that, various exemplary logic blocks, modules, circuits, and algorithm steps described with reference to the disclosure herein may be implemented as specialized electronic hardware, computer software, or a combination of electronic hardware and computer software. For examples, the modules/units may be implemented by a processor executing software instructions stored in the computer-readable storage medium.
The flowcharts and block diagrams in the accompanying drawings show system architectures, functions, and operations of possible implementations of the system and method according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent one module, one program segment, or a part of code, where the module, the program segment, or the part of code includes one or more executable instructions used for implementing specified logic functions. It should also be noted that, in some alternative implementations, functions marked in the blocks may also occur in a sequence different from the sequence marked in the drawing. For example, two consecutive blocks may be substantially executed in parallel, and sometimes, they may also be executed in the reverse order, depending on the functions involved. Each block in the block diagram and/or flowchart, and a combination of blocks in the block diagram and/or flowchart, may be implemented by a dedicated hardware-based system for executing corresponding functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.
It is appreciated that embodiments of the present disclosure may be embodied as a method, a system or a computer program product. Accordingly, embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware for allowing specialized components to perform the functions described above. Furthermore, embodiments of the present disclosure may take the form of a computer program product embodied in one or more computer-readable storage media (including but not limited to a magnetic disk memory, a CD-ROM, an optical memory, etc.) containing computer-readable program codes.
Embodiments of the present disclosure are described with reference to flow diagrams and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a computer, an embedded processor, or other programmable data processing devices to produce a machine, such that the instructions, which are executed via the processor of the computer or other programmable data processing devices, create a means for implementing the functions specified in one or more flows in the flow diagrams and/or one or more blocks in the block diagrams.
These computer program instructions may also be stored in a computer-readable memory that may direct a computer or other programmable data processing devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce a manufactured product including an instruction means that implements the functions specified in one or more flows in the flow diagrams and/or one or more blocks in the block diagrams.
These computer program instructions may also be loaded onto a computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce processing implemented by the computer, such that the instructions which are executed on the computer or other programmable devices provide steps for implementing the functions specified in one or more flows in the flow diagrams and/or one or more blocks in the block diagrams. In a typical configuration, a computer device includes one or more Central Processing Units (CPUs), an input/output interface, a network interface, and a memory. The memory may include forms of a volatile memory, a random access memory (RAM), and/or non-volatile memory and the like, such as a read-only memory (ROM) or a flash RAM in a computer-readable storage medium. The memory is an example of the computer-readable storage medium.
The computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The computer-readable medium includes non-volatile and volatile media, and removable and non-removable media, wherein information storage may be implemented with any method or technology. Information may be modules of computer-readable instructions, data structures and programs, or other data. Examples of a non-transitory computer-readable medium include but are not limited to a phase-change random access memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memories (RAMs), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or other memory technologies, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD) or other optical storage, a cassette tape, tape or disk storage or other magnetic storage devices, a cache, a register, or any other non-transmission media that may be used to store information capable of being accessed by a computer device. The computer-readable storage medium is non-transitory, and does not include transitory media, such as modulated data signals and carrier waves.
The illustrated steps are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. Thus, these examples are presented herein for purposes of illustration, and not limitation. For example, steps or processes disclosed herein are not limited to being performed in the order described, but may be performed in any order, and some steps may be omitted, consistent with the disclosed embodiments. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries may be defined so long as the specified functions and relationships thereof are appropriately performed. It is appreciated that, alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) based on the teachings fall within the scope and spirit of the disclosed embodiments.
While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. Also, the words “comprising”, “having”, “containing”, and “including,” and other similar forms are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms “a”, “an”, and “the” include plural references unless the context clearly dictates otherwise.
It will be appreciated that the present invention is not limited to the exact construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. It is intended that the scope of the invention should only be limited by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| CN 201610574955.7 | Jul 2016 | CN | national |
