Patent Grant
11114045
A seamless two-stage method of enhancing the image displayed on an LCD screen in a high ambient light condition or environment.
Information displays are generally difficult to view or distinguish in direct sunlight or other high ambient light conditions due to insufficient emitted light and contrast.
Image enhancement is widely used in a wide array of endeavors such as medical image analysis, remote sensing, industrial X-ray image processing and microscopic imaging. Image enhancement is employed to improve the visual effects and the clarity of images for more effective results.
Contrast enhancement is one of the commonly used image enhancement methods. Many methods for image contrast enhancement have been published and are widely known. One of the most common techniques is histogram equalization. The fundamental principle of histogram equalization is to process the image such that the enhanced image has an approximately uniform histogram distribution resulting in the dynamic range of the image being fully exploited. This process can enhance the visibility of imagery. However, these techniques are not suitable for all images causing several problems such as:
Another technique to improve the viewing of images in high ambient light when the display's inherent visual power is inadequate for the conditions is to increase brightness globally throughout the image by applying an image filter or through hardware means on the display itself. In transmissive type LCD displays, this results in a generalized increase in the transparency of the liquid crystals allowing more of the display's backlighting to pass through to the viewer. Under some conditions this process may enhance the visibility of imagery. However, this technique is not suitable for all images causing problems, primarily because the technique introduces a reduction of dynamic range in the image leading to low contrast in a way that is contrary to the goal of enhancing the visibility of detail.
In order to overcome these short comings an adaptive image processing is employed by this invention to automatically adjust a tonality transformation of the image according to an analysis of the object image's histogram.
Examples of the prior art are found in the following references;
“A New Enhancement Approach for Enhancing Image of Digital Cameras by Changing Contract” International Journal of Advanced Science and Technology—Vol. 32, July, 2011 http://www.sersc.org/journals/DAST/vol32/2.pdf
“An Adaptive Image Enhancement Technique Preserving Brightness Level Using Gamma Correction” Advance in Electronic and Electric Engineering. ISSN 2231-1297, Volume 3, Number 9 (2013), pp. 1097-1108 http://www.ripublication.com/aeee/060_pp %201097-1108.pdf
“A Novel Method for the Contrast Enhancement of Fog Degraded Video Sequences” International Journal of Computer Applications (0975-8887) Volume 54-No. 13, September 2012 http://research.ijcaonline.org/volume54/number13/pxc3882489.pdf
“System and method for enhancing low-visibility imagery” Publication U.S. Pat. No. 8,023,760 B1 Sep. 20, 2011-Assignee: US Navy http://www.google.com/patents/U.S. Pat. No. 8,023,760
“Adaptive linear contrast method for enhancement of low-visibility imagery” Publication U.S. Pat. No. 8,149,245 B1 Apr. 3, 2012-Assignee: US Navy https://www.google.com/patents/U.S. Pat. No. 8,149,245
“Image Processing for Human Understanding in Low-visibility” Mark A. Livingston, Caelan R. Garrett, and Zhuming Ai—Naval Research Laboratory http://web.mit.edu/caelan/www/publications/hsis2011.pdf
“Color Image Segmentation: Advances & Prospects”, H. D. Cheng, X. H. Jiang, Y. Sun and Jing Li Wang; Dept. of Computer Science, Utah State University, Logan, Utah 84322-4205 describes a summary of color image segmentation techniques. These techniques maybe based on monochrome segmentation operating for segmenting monochrome images: histogram thresholding, characteristic feature clustering, edge detection, region-based methods, fuzzy techniques and neural networks. A review of major color representation methods and their advantages/disadvantages and finally summarize the color image segmentation techniques using different color representations are also included. The usage of color models for image segmentation and novel approaches such as fuzzy method and physics based method are also discussed.
US 2010/0053222 discloses systems and methods for modifying or adjusting a display source light illumination level based on power consumption goals. In some embodiments, a rate control parameter is used to limit the rate at which the illumination level is varied. In some embodiments, image content analysis may be used to determine the value of the rate control parameter.
U.S. Pat. No. 7,995,116 relates to a method and digital camera to capture an initial set of evaluation images. A plurality of characteristics of the initial set of evaluation images are then assessed. The characteristics include subject motion between the initial set of evaluation images. When the subject motion is in excess of a predetermined threshold, a final capture state of the camera is set responsive to the initial assessment. When the subject motion is less than the predetermined threshold, the evaluation images are analyzed to provide analysis results and the final capture state of the camera is set responsive to the initial assessment and the analysis results.
US 2012/0092393 teaches techniques to dynamically regulate brightness of backlighting in display devices. The brightness dynamic range of a display device is adjusted according to the current ambient viewing conditions. In order words, the original brightness dynamic range of the display image is mapped to the brightness dynamic range suitable for human eyes under the current ambient viewing conditions. The brightness of the display image is corrected according to a histogram to enhance the contrast and details of the display image, thereby a high-quality displayed image can be presented under the current ambient viewing conditions.
U.S. Pat. No. 8,203,579 describes systems and methods for selecting a display source light illumination level and temporal filtering of the display source light illumination level.
Adobe Photoshop, a well-known and powerful image editing tool, has a full suite of image processing filters. Included is a tool, they call “Curves” for transforming brightness and is presented to the user in a similar manner, with histogram and transformation function graph, as shown in this invention discloser. Photoshop features an “Auto” function that manipulates the transformation function to enhance the image.
In review, each of these processes or techniques has some similarities in that they are adaptive but none process images in the manner of the present invention or any possess the range of adaption necessary for presenting a wide variety of imagery with useful detail, particularly in high ambient light conditions.
The present invention relates to a method and system for enhancing the visibility of object images that may otherwise be unsuitable for display given the image's points of interest and the viewing conditions thereby enhancing the process of extracting and presenting useful information.
The image enhancement of the present system adaptively enhances the visibility of low contrast and/or low brightness images. The system processes an input image so that the resultant image is more suitable than the original image for a specific application such as the viewing of pertinent details of low visibility images in a high ambient light environment. Instead of simply making the back-light brighter, the present invention transforms images to allow sufficient transmissive light through the assembly from the back-light as well of enhancing viewability by increasing contrast for sunlight viewing.
In particular, the present invention relates to:
The adaptive transformation of image tonality employed by this invention selectively increases contrast, brightness and color saturation in such a way as to mimic increased display power. Displays, such as those found on most smart phones, tablets, computers, televisions, marine and avionics instrumentations, have a finite amount of display brightness and contrast. This invention may be applied to effectively extend display power well beyond the physical limitations of the display. For instance, by coupling the ambient light sensor signal used to adjust the backlighting of a LCD screen to this invention, it is possible to vary this invention's image processing strength to the effect of seamlessly increasing visibility of the display after the backlighting of the device has reached its maximum.
For a fuller understanding of the nature and object of the invention, reference should be had to the following detailed description taken in connection with the accompanying drawings in which:
Similar reference characters refer to similar parts throughout the several views of the drawings.
The present invention relates to a seamless image processing method and system to adaptively increase the visibility of details in low contrast and/or low brightness images and to boost the visual clarity in otherwise inadequate images when shown on underpowered displays and/or high ambient light conditions in a way that mimics increased display power.
In particular, the console 10 comprises an enclosure 20 to house an electronics assembly including a micro-controller and circuitry 22 coupled to an external power and data source (not shown) and to an ambient light sensor 24 and back-light 26 comprising a plurality of LEDs each indicated as 28. Also housed within the enclosure 20 is an LCD display 30 disposed to receive the light emitted from the LEDs 28 and a protective transparent panel or cover of glass 32. A touch screen 34 may be operatively disposed between the LCD display 30 and the protective transparent panel or cover glass 32.
The apparatus shown in
The second stage of the seamless image processing system of the present invention comprises a method having the plurality of steps including image analysis, transformation calculation and image transformation as depicted in a high-level view by
Image Analysis Step
The image analysis step involves the derivation of several characteristics of the object image. The image analysis may be performed on a pixel-by-pixel basis of the object image or, optionally, sampled more sparsely. Image characteristics derived are:
Grey scale values for each pixel of the object image are calculated using one of many commonly known methods. For example:
The grey scale frequency histogram is a statistical representation of the distribution of grey scale values found in the object image. It is a representation of how many of the pixels or sample points fall into a range of predefined values called bins. As the image is scanned each sample point's grey value increments the appropriate accumulator resulting in a depiction of how many sample point grey values where found in the range of each bin. In the image processing examples to follow 256 bins are used in the histogram where 0 represents no brightness (black) and 255 represents maximum brightness (white).
Pseudocode for Grey Scale Histogram using Numerical Brightness and 256 bins on a 24-bit depth image:
array histogramBin[256]
for j=0 to imageWidth {
The average brightness is calculated by converting each of the target image pixel values to grey scale brightness and calculating the arithmetic mean.
Pseudocode for Average Brightness using Numerical Brightness and Arithmetic Mean:
accumulatedBrightness=0
for j=0 to imageWidth
result=accumulatedBrightness/(imageWidth*imageHeight) Image Range of Brightness
The range of brightness is defined by this invention as a measure of brightness extremes found in the image. Continuing with the example of a 24-bit image depth and an 8-bit valuation ranging from 0-255; 0 representing no range; i.e. all pixels are the same brightness, whereas, 255 represents that the full dynamic range of brightness is expressed by the object image.
The simplest means of calculating this is to scan thought each pixel, converting to brightness, and recording the lowest brightness pixel and the highest brightness pixel value found and then taking the difference.
Pseudocode for Simple Range of Brightness:
A major drawback of Simple Range of Brightness is that if the image has the majority of its pixels in a narrow range of brightness and just a few pixels at the extremes (in this case it only takes one pixel) the resulting range value is not representative of the image as a whole.
To overcome this limitation many modifications are available, such as:
The first method reduces the drawback of Simple Range of Brightness but it relies on the careful selection of the threshold value to get optimal performance and there is no value that is best for all images.
The second method is preferred because it does not rely on any arbitrary threshold value.
Pseudocode for Divided Range of Brightness:
The invention's image process involves processing the object image according to a transformation function that maps image tonality adjustments. This transformation can be visualized in graph form as a line that expresses the input and output values of the function. When the function is at unity its input value are the output values and the transformation is neutral in that no image modification occurs as
Performing a neutral transformation on image brightness has no effect. Each pixel's input brightness maps to the same output brightness value (exp. input 127 results in output 127.)
The focus of this invention involves automatically modifying this transformation curve to maximize detail in the brightness regions that contain the majority of detail while maintaining the full dynamic range of the image; i.e. black remains black and white remains white. Maximum detail is realized by insuring the slope of the transformation curve is steep through the regions of highest histogramic brightness frequency density. This is accomplished by integrating the histogram and normalizing the resultant values to the brightness transformation range, in this example 0 to 255.
After the application of the histogram integration and normalization the modification of the transformation curve is shown in
The integration of the histogram data can be straight forward or enhanced to provide greater amplification of the more subtle details in the image. Enhancement may involve increased weighting of histogram values when the histogram values are on the rise and/or increased weighing of darker values over light values or other integration like functions or equivalents thereof.
Pseudocode for Basic Integration:
Pseudocode for One Example of Enhanced Integration Boosting Detail of Darker Image Features:
Pseudocode for Normalization of Histogram Integration Brightness Transformation Data to a Range of 0 to 255:
This invention's image processing involves processing the object image according to the results of the Brightness Transformation, Average Brightness and Range of Brightness values derived from the object image as shown in
Image Transformation is where the previous Image Analysis and Transformation Calculations are applied to the object image to produce the output image result. Each pixel is broke down into its red [R], green [G] and blue [B] intensity values, mathematically manipulated and then recombined into the new resulting pixel value. The example algorithm to follow utilizes the previous Image Analysis and Transformation Calculations to manipulate the brightness, contrast and color saturation to effect the desired result. The optional effectIntensity variable provides a means for external control of the intensity of the transformation's effect on the resulting image. It may be used to input a display device ambient light sensor to extend the effective display power beyond the display's native physical limitation in response to lighting conditions. The nominal value for full effect in this example is 100. A value of 0 results in no effect leaving the image unaltered. The value is continuously variable and open ended and can be thought of as a percent where 150 represents 150% effect, for example.
Pseudocode for Image Transformation utilizing Image Analysis and Transformation Calculations Data:
Six (6) examples of original input images, corresponding histograms and brightness transformation curves and resulting processed images are shown. In all examples this invention's effect is set at 100% with no manual adjustments introduced from image to image demonstrating its adaptive nature.
This is a continuation application of utility nonprovisional application Ser. No. 15/330,246, filed Aug. 29, 2016 that claims priority of provisional application No. 62/283,421, filed Aug. 31, 2015.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20030020974 | Matsushima | Jan 2003 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 62283421 | Aug 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15330246 | Aug 2016 | US |
| Child | 16602528 | US |
