TECHNICAL FIELD
The present disclosure relates generally to digital images and/or videos, and in particular, to enhancing quality of images and/or videos that are captured in low light environments.
Generally speaking, capturing videos and/or images in low-light environments is challenging. One possible solution for capturing single images in low-light conditions is to use a flash. However, it may not be possible to use a strong light source (such as a flash) while capturing videos in low-light environments. The reason might be that the strong light source can drain battery of the video-camera very quickly. There is a need in the art for techniques to enable a device to capture videos and/or images in low-light conditions.
Certain embodiments present a method for enhancing quality of a first image that is captured in a low-light environment. The method generally includes, in part, generating a second image by brightening a plurality of pixels in the first image based on a predefined criteria, and generating a third image using an edge-preserving noise reduction algorithm based on the second image. In one embodiment, the predefined criteria includes one or more look-up tables for mapping at least one of color or brightness values of the first image to the second image.
In one embodiment, the method further includes, generating a composite image by calculating a weighted average of the first image and the third image. In one embodiment, the method includes, calculating at least one weight corresponding to each pixel based on average intensity in a neighborhood around the pixel in the first image. The at least one weight is used in the weighted average of the first image and the third image. The at least one weight has a value between zero and one, which is calculated based on a monotonically increasing function of the average intensity in the neighborhood around the pixel.
In one embodiment, the at least one weight is calculated for pixels taken from a blurred and/or downsized version of the first image. For certain embodiments, the method is applied on a plurality of first images and the one or more look-up tables are adapted for each of the plurality of first images based on brightness of an input scene.
In one embodiment, generating the third image includes, in part, generating an edge map of the second image, and generating the third image by obtaining a weighted average of the second image and a fourth image based at least on the edge map. The fourth image is generated by blurring the second image. In one embodiment, for each pixel in the second image, a weight corresponding to the second image is determined based on the edge map. The weight is larger than 0.5 if the pixel represents an edge, and is smaller than 0.5 if the pixel is part of a smooth area in the second image.
Certain embodiments present an apparatus for enhancing quality of a first image that is captured in a low-light environment. The apparatus generally includes, in part, means for generating a second image by brightening a plurality of pixels in the first image based on a predefined criteria, and means for generating a third image using an edge-preserving noise reduction algorithm based on the second image.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
An understanding of the nature and advantages of various embodiments may be realized by reference to the following figures. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.
Certain embodiments present a technique for enhancing quality of videos and/or images that are captured in low-light environments by processing the captured video frames and/or images. In this document, the term “image” is used to refer to a photograph captured by a camera, each of the frames in a video, and/or any other type of visual data captured from a scene, all of which fall within the teachings of the present disclosure.
In one embodiment, quality of an image is improved by brightening and/or de-noising the image. Quality of an image may refer to the visual perception of the image in terms of its sharpness and the level of detail that is visible in the image. Looking into the nature. Owls can see details of their surroundings in dark. One of the motivations of the present disclosure is to develop a technique that does not require a strong light source (such as a flash) for capturing images and/or videos in low-light environments with reasonable quality (similar to the processing that may be done in an Owl's eye).
Generally, it may be difficult to resolve detail in a video that is captured in low-light environments. One possible solution to process the captured video frames and brighten up each of the dark pixels. However, brightening often results in an undesirable magnification of noise (including, for example, quantization noise). Another possible solution is to use video cameras that are very sensitive to light (e.g., High ISO videos). However, High ISO videos may also result in high noise levels.
Other brightening approaches include histogram equalization of video frames. However, using histogram equalization results in a washed-out image with noise significantly boosted. Another possible approach is to use a special sensor that is able to capture more light in a shorter exposure time. For example, instead of capturing a video with red-green-blue-green (RGBG) Bayer pattern, one may capture a video using red-green-blue-clear (RGBC), or RCBC. However, this approach increases cost of the video cameras.
One embodiment brightness one or more dark pixels in an image based on a criteria. The dark pixels are identified by analyzing brightness of each pixel. In one embodiment, each pixel in the image is brightened depending on the original level of brightness in the pixel. For example, a pixel that is originally dark may be brightened more compared to a pixel that is originally bright. In one embodiment, the image is brightened such that a non-decreasing continuity and a reasonable level of contrast is maintained in the brightened image.
In one embodiment, one or more look-up tables are defined for brightening the input images. As an example, a length-256 brightening lookup table is calculated to map each input integer value between 0 and 255 to a brightened value based on a predefined rule. In general, the look up tables can be defined in advance and stored on the device. Alternatively, brightness values corresponding to each brightened pixel can be calculated based on a formula.
For certain embodiments, the look-up tables can be adapted on a frame-by-frame basis in a video according to the brightness of the input scene. For example, if a scene from which a video is captured is sufficiently bright, the pixels in the video may not need to be brightened (or may need a little brightening depending on the original level of light in the environment). On the other hand, if the scene is originally dark, the pixels in the video need more brightening.
For certain embodiments, similar or different look-up tables can be defined corresponding to different channels. For example, one or more look up tables can be defined corresponding to each of the channels in RGB color space, the Y channel (e.g., brightness channel) in YUV color space, or the V channel in HSV model.
In the present disclosure, YUV and RGB (red-green-blue) refer to color spaces representing information about each pixel in the image. In addition, HSV (Hue-Saturation-Value) refers to a cylindrical-coordinate representation of points in the RGB color model. It should be noted that the present disclosure is not limited to any specific representation. One of ordinary skill in the art would readily understand that the image quality enhancing technique as described herein can be applied to any type of representation without departing from the scope of the present disclosure.
In one embodiment, a weighted combination of the original and brightened frames may be generated. The weights corresponding to each of the original and the brightened frames are defined based on the original frame in order to adapt the output frames' brightness intensities.
In one embodiment, a brightened and de-noised image 216 is generated by applying the edge-preserving noise reduction technique to the brightened image 204, and a blurred version of the brightened image (e.g., brightened-blurred image 206). For example, a pixel-wise weighted average 212 of the brightened image 204 and the brightened-blurred image 206 is calculated. The weighted average is constructed for each pixel coordinate (x,y), as follows:
I
4(x,y)=W1(x,y)×I2(x,y)+(1−W1(x,y))×I3(x,y),
where I2 is the brightened image, I3 is the brightened-blurred image, I4 is the resulting brightened-denoised image, and W1 is the de-noising mixing mask with values between zero and one.
In one embodiment, the de-noising mixing mask W1 is obtained as a function of the edge-map values. As an example, higher edge-map values (e.g., representing an edge) correspond to higher de-noising mixing mask values. The edge-map 210 can be generated based either on the original image 202 or the brightened image 204. In one embodiment, the edge-map 210 is generated by applying a blurred magnitude of a difference of Gaussian filter to the image. In one embodiment, the difference of Gaussian operation may be computed on a down-sampled (e.g., downsized) version of the image, or on the image itself.
As an example, for the pixels that are located on the edges, the brightened image 204 is given a higher weight than the blurred version 206 of the brightened image (e.g., 0.5<W1≦1). In addition, the weights are selected such that the pixels in the smooth regions are chosen from the brightened-blurred image (e.g., 0≦W1<0.5). Therefore, for the pixels that are located on the smooth regions, the blurred version of the brightened image is given a higher weight than the brightened image. In addition, for the intermediate pixels (e.g., pixels between the sharp edges and the smooth regions), the weights are chosen such that these pixels represent a weighted average between the corresponding pixels in the brightened image 204 and the brightened-blurred image 206. For example, the weights may increase linearly for the pixels that are between the sharp edges and the smooth regions. Therefore, by moving away from an edge, weight of the brightened image decreases and weight of the brightened-blurred image increases. It should be noted linearly increasing weights are mentioned only as an example, and any other relation may be defined for generating the weights without departing from the scope of the present disclosure.
As mentioned earlier, the noise reduction technique as described herein preserves edges in the image while reducing noise. Therefore, unlike other noise reduction schemes in the art, the image does not appear washed-out after noise reduction. In one variation of the noise reduction technique, the weights can be generated such that the convex combination of the brightened image and the brightened-blurred image favors the sharp version (e.g., the brightened image) over the brightened-blurred image in brighter regions (e.g., where noise reduction is not as necessary).
It should be noted
In some cases, brightening and de-noising an image may cause unwanted artifacts. Therefore, in one embodiment, the brightened-denoised image may be adjusted to revert back towards the original image, if the original image had an acceptable and/or better quality. Certain embodiments selectively brighten an image based on a brightness map 214 of the original image using a local tone mapping (LTM) technique. The LTM technique adjusts brightness of each pixel in an image by leveraging sharpness of the original image to generate a composite image 220. The brightness map 214 may be obtained, for example, by blurring the original image.
In one embodiment, the composite image 220 is generated as a pixel wise weighted average 218 between the original image 202 and the brightened-denoised image 216, as follows:
I
5(x,y)=W2(x,y)×I1(x,y)+(1−W2(x,y))×I4(x,y),
where I1 is the original image, I4 is the brightened-denoised image, I5 is the resulting composite image, and W2 is the LTM mixing mask. In one embodiment, the LTM mixing mask is computed as a function of brightness of the image (e.g., based on the brightness map values). The LTM mixing mask has values between zero and one, as illustrated in
In one embodiment, the LTM mixing mask W2 for a pixel may be close to one if the average intensity in a neighborhood around the corresponding pixel in the brightened image is high. In addition, the LTM mixing mask W2 may be close to zero if the average intensity in a neighborhood around the corresponding pixel in the brightened image is low. In general, the LTM mixing mask may be described as a lookup-table and/or as a function of brightness of a pixel.
In one embodiment, an LTM mixing mask is generated by blurring the original image. For example, the original image is blurred with a sigma equal to width of the image divided by 10 to generate a blurred image. Next, the LTM mixing mask is constructed from the blurred image based on the brightness of each pixel. As described earlier, the LTM mixing mask is used to merge the original image with the brightened-denoised image. For certain embodiments, the original image and the brightened-denoised image are merged either in one channel (e.g., Y channel), or in more than one channel (e.g., Luma and/or color channels). In one embodiment, the blurred image may be down-sampled to a smaller width (e.g., 128) to facilitate further processing, such as additional blurring and/or generating the look up table for the LTM mixing mask.
In one embodiment, the device generates an edge-map (e.g., using difference of Gaussian filter) from the second (e.g., brightened) image. In another embodiment, the edge-map is generated from the first image (e.g., the original image). The edge-map may show where the edges of objects in the scene are located.
Techniques for enhancing quality of images and/or videos captured in low-lights environments are described. In one embodiment, an image is brightened and de-noised such that the details in the image are more visible, without increasing the noise level. As a result, one may capture videos in low-light environments and process them later to enhance their quality. These techniques enable capturing low cost videos and/or images in low light environments.
In the embodiment shown at
Memory 720 may be coupled to processor 710. In some embodiments, memory 720 offers both short-term and long-term storage and may in fact be divided into several units. Short term memory may store images which may be discarded after an analysis. Alternatively, all images may be stored in long term storage depending on user selections. Memory 720 may be volatile, such as static random access memory (SRAM) and/or dynamic random access memory (DRAM) and/or non-volatile, such as read-only memory (ROM), flash memory, and the like. Furthermore, memory 720 can include removable storage devices, such as secure digital (SD) cards. Thus, memory 720 provides storage of computer readable instructions, data structures, program modules, and other data for mobile device 700. In some embodiments, memory 720 may be distributed into different hardware modules.
In some embodiments, memory 720 stores a plurality of applications 726. Applications 726 contain particular instructions to be executed by processor 710. In alternative embodiments, other hardware modules may additionally execute certain applications or parts of applications. Memory 720 may be used to store computer readable instructions for modules that implement scanning according to certain embodiments, and may also store compact object representations as part of a database.
In some embodiments, memory 720 includes an operating system 723. Operating system 723 may be operable to initiate the execution of the instructions provided by application modules anchor manage other hardware modules as well as interfaces with communication modules which may use wireless transceiver 712 and a link 716. Operating system 723 may be adapted to perform other operations across the components of mobile device 700, including threading, resource management, data storage control and other similar functionality.
In some embodiments, mobile device 700 includes a plurality of other hardware modules 701. Each of the other hardware modules 701 is a physical module within mobile device 700. However, while each of the hardware modules 701 is permanently configured as a structure, a respective one of hardware modules may be temporarily configured to perform specific functions or temporarily activated.
Other embodiments may include sensors integrated into device 700. An example of a sensor 762 can be, for example, an accelerometer, a Wi-Fi transceiver, a satellite navigation system receiver (e.g., a GPS module), a pressure module, a temperature module, an audio output and/or input module (e.g., a microphone), a camera module, a proximity sensor, an alternate line service (ALS) module, a capacitive touch sensor, a near field communication (NFC) module, a Bluetooth transceiver, a cellular transceiver, a magnetometer, a gyroscope, an inertial sensor (e.g., a module the combines an accelerometer and a gyroscope), an ambient light sensor, a relative humidity sensor, or any other similar module operable to provide sensory output and/or receive sensory input. In some embodiments, one or more functions of the sensors 762 may be implemented as hardware, software, or firmware. Further, as described herein, certain hardware modules such as the accelerometer, the GPS module, the gyroscope, the inertial sensor, or other such modules may be used in conjunction with the camera and image processing module to provide additional information. In certain embodiments, a user may use a user input module 706 to select how to analyze the images.
Mobile device 700 may include a component such as a wireless communication module which may integrate antenna 718 and wireless transceiver 712 with any other hardware, firmware, or software necessary for wireless communications. Such a wireless communication module may be configured to receive signals from various devices such as data sources via networks and access points such as a network access point. In certain embodiments, compact object representations may be communicated to server computers, other mobile devices, or other networked computing devices to be stored in a remote database and used by multiple other devices when the devices execute object recognition functionality.
In addition to other hardware modules and applications in memory 720, mobile device 700 may have a display output 703 and a user input module 706. Display output 703 graphically presents information from mobile device 700 to the user. This information may be derived from one or more application modules, one or more hardware modules, a combination thereof, or any other suitable means for resolving graphical content for the user (e.g., by operating system 723). Display output 703 can be liquid crystal display (LCD) technology, light emitting polymer display (LPD) technology, or some other display technology. In some embodiments, display module 703 is a capacitive or resistive touch screen and may be sensitive to haptic and/or tactile contact with a user. In such embodiments, the display output 703 can comprise a multi-touch-sensitive display. Display output 703 may then be used to display any number of outputs associated with a camera 721 or image processing module 722, such as alerts, settings, thresholds, user interfaces, or other such controls.
The methods, systems, and devices discussed above are examples. Various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, in alternative configurations, the methods described may be performed in an order different from that described, and/or various stages may be added, omitted, and/or combined. Also, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner.
Specific details are given in the description to provide a thorough understanding of the embodiments. However, embodiments may be practiced without certain specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been mentioned without unnecessary detail in order to avoid obscuring the embodiments. This description provides example embodiments only, and is not intended to limit the scope, applicability, or configuration of various embodiments. Rather, the preceding description of the embodiments will provide those skilled in the art with an enabling description for implementing embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of various embodiments.
Also, some embodiments were described as processes which may be depicted in a flow with process arrows. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Furthermore, embodiments of the methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the associated tasks may be stored in a computer-readable medium such as a storage medium. Processors may perform the associated tasks. Additionally, the above elements may merely be a component of a larger system, wherein other rules may take precedence over or otherwise modify the application of various embodiments, and any number of steps may be undertaken before, during, or after the elements of any embodiment are implemented.
It should be noted that the method as described herein may be implemented in software. The software may in general be stored in a non-transitory storage device (e.g., memory) and carried out by a processor (e.g., a general purpose processor, a digital signal processor, and the like.)
Having described several embodiments, it will therefore be clear to a person of ordinary skill that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the disclosure.
The present application claims priority to Provisional Application. No. 61/872,588, entitled “Method and Apparatus for enhancing low-light videos,” filed Aug. 30, 2013, and Provisional Application No. 61/937,787, entitled “Method and Apparatus for enhancing low-light videos,” filed Feb. 10, 2014, both of which are assigned to the assignee hereof and expressly incorporated by reference herein in their entirety.
Number | Date | Country | |
---|---|---|---|
61872588 | Aug 2013 | US | |
61937787 | Feb 2014 | US |