Patent Application
20090274383
The subject application is generally directed to enhancement of digitally encoded images. The application is particularly suited to automatically improving an appearance of rendered electronic images by selectively stretching a black level associated with image data.
Electronic images include those images acquired by digital camera, or other image capture or acquisition systems. Often times acquired images appear to be more washed out or lacking in definition given the circumstances during which an image was obtained. Factors affecting image appearance includes lighting levels, lighting position, balance of colors, and proximity of objects and size of objects in a captured image. Many acquired images can be improved by adjusting various image properties through software-based data manipulation. Applications, such as Adobe Photoshop, include controls such as a slider control which enables a user to manually adjust a brightness or darkness of an image by adjustment of its image data, typically stored as in a multidimensional color space such as RGB, CMYK, or any other multidimensional color system.
While image adjustment by brightness control can improve a rendered image, there is substantial opportunity for inexact adjustment of brightness or darkness given the parameters that are adjustable and that humans are involved in the process.
In accordance with one embodiment of the subject application, there is provided a system and method for enhancement of digitally encoded images.
Further in accordance with one embodiment of the subject application, there is provided a system and method for automatically improving an appearance of rendered electronic images by selectively stretching a black level associated with image data.
Still further in accordance with one embodiment of the subject application, there is provided a global darkness image enhancement system. The system comprises means adapted for acquiring image data encoded in a multidimensional color space and means adapted for calculating histogram data in accordance with acquired image data. The system also comprises means adapted for detecting a ramp zone associated with calculated histogram data and enhancement means adapted for selectively stretching a black level associated with the acquired image data in accordance with a detected ramp zone so as to generate enhanced image data. The system further comprises means adapted for outputting enhanced image data to at least one of an associated data storage and an associated display.
Further in accordance with one embodiment of the subject application, there is provided a global darkness image enhancement method. The method includes the step of acquiring image data encoded in a multidimensional color space. The method also includes the step of calculating histogram data in accordance with acquired image data. The method further comprises the steps of detecting a ramp zone associated with the calculated histogram data, and selectively stretching a black level associated with the acquired image data in accordance with a detected ramp zone so as to generate enhanced image data. The method also comprises the step of outputting enhanced image data to at least one of an associated data storage and an associated display.
Still other advantages, aspects and features of the subject application will become readily apparent to those skilled in the art from the following description wherein there is shown and described a preferred embodiment of the subject application, simply by way of illustration of one of the best modes best suited to carry out the subject application. As it will be realized, the subject application is capable of other different embodiments and its several details are capable of modifications in various obvious aspects all without departing from the scope of the subject application. Accordingly, the drawings and descriptions will be regarded as illustrative in nature and not as restrictive.
The subject application is described with reference to certain figures, including:
The subject application is directed to a system and method for enhancement of digitally encoded images. In particular, the subject application is directed to a system and method for automatically improving an appearance of rendered electronic images by selectively stretching a black level associated with image data. More particularly, the subject application is directed to a system and method for global darkness image enhancement. It will become apparent to those skilled in the art that the system and method described herein are suitably adapted to a plurality of varying electronic fields employing image enhancement, including, for example and without limitation, communications, general computing, data processing, document processing, or the like. The preferred embodiment, as depicted in
Referring now to
The system 100 also includes a document processing device 104, depicted in
According to one embodiment of the subject application, the document processing device 104 is suitably equipped to receive a plurality of portable storage media, including, without limitation, Firewire drive, USB drive, SD, MMC, XD, Compact Flash, Memory Stick, and the like. In the preferred embodiment of the subject application, the document processing device 104 further includes an associated user interface 106, such as a touch-screen, LCD display, touch-panel, alpha-numeric keypad, or the like, via which an associated user is able to interact directly with the document processing device 104. In accordance with the preferred embodiment of the subject application, the user interface 106 is advantageously used to communicate information to the associated user and receive selections from the associated user. The skilled artisan will appreciate that the user interface 106 comprises various components, suitably adapted to present data to the associated user, as are known in the art. In accordance with one embodiment of the subject application, the user interface 106 comprises a display, suitably adapted to display one or more graphical elements, text data, images, or the like, to an associated user, receive input from the associated user, and communicate the same to a backend component, such as a controller 108, as explained in greater detail below. Preferably, the document processing device 104 is communicatively coupled to the computer network 102 via a suitable communications link 112. As will be understood by those skilled in the art, suitable communications links include, for example and without limitation, WiMax, 802.11a, 802.11b, 802.11g, 802.11(x), Bluetooth, the public switched telephone network, a proprietary communications network, infrared, optical, or any other suitable wired or wireless data transmission communications known in the art.
In accordance with one embodiment of the subject application, the document processing device 104 further incorporates a backend component, designated as the controller 108, suitably adapted to facilitate the operations of the document processing device 104, as will be understood by those skilled in the art. Preferably, the controller 108 is embodied as hardware, software, or any suitable combination thereof, configured to control the operations of the associated document processing device 104, facilitate the display of images via the user interface 106, direct the manipulation of electronic image data, and the like. For purposes of explanation, the controller 108 is used to refer to any myriad of components associated with the document processing device 104, including hardware, software, or combinations thereof, functioning to perform, cause to be performed, control, or otherwise direct the methodologies described hereinafter. It will be understood by those skilled in the art that the methodologies described with respect to the controller 108 are capable of being performed by any general purpose computing system, known in the art, and thus the controller 108 is representative of such a general computing device and is intended as such when used hereinafter. Furthermore, the use of the controller 108 hereinafter is for the example embodiment only, and other embodiments, which will be apparent to one skilled in the art, are capable of employing the system and method for global darkness image enhancement of the subject application. The functioning of the controller 108 will better be understood in conjunction with the block diagrams illustrated in
Communicatively coupled to the document processing device 104 is a data storage device 110. In accordance with the preferred embodiment of the subject application, the data storage device 110 is any mass storage device known in the art including, for example and without limitation, magnetic storage drives, a hard disk drive, optical storage devices, flash memory devices, or any suitable combination thereof. In the preferred embodiment, the data storage device 110 is suitably adapted to store document data, image data, electronic database data, or the like. It will be appreciated by those skilled in the art that while illustrated in
The system 100 illustrated in
Turning now to
Also included in the controller 200 is random access memory 206, suitably formed of dynamic random access memory, static random access memory, or any other suitable, addressable and writable memory system. Random access memory provides a storage area for data instructions associated with applications and data handling accomplished by processor 202.
A storage interface 208 suitably provides a mechanism for non-volatile, bulk or long term storage of data associated with the controller 200. The storage interface 208 suitably uses bulk storage, such as any suitable addressable or serial storage, such as a disk, optical, tape drive and the like as shown as 216, as well as any suitable storage medium as will be appreciated by one of ordinary skill in the art.
A network interface subsystem 210 suitably routes input and output from an associated network allowing the controller 200 to communicate to other devices. The network interface subsystem 210 suitably interfaces with one or more connections with external devices to the device 200. By way of example, illustrated is at least one network interface card 214 for data communication with fixed or wired networks, such as Ethernet, token ring, and the like, and a wireless interface 218, suitably adapted for wireless communication via means such as WiFi, WiMax, wireless modem, cellular network, or any suitable wireless communication system. It is to be appreciated however, that the network interface subsystem suitably utilizes any physical or non-physical data transfer layer or protocol layer as will be appreciated by one of ordinary skill in the art. In the illustration, the network interface 214 is interconnected for data interchange via a physical network 220, suitably comprised of a local area network, wide area network, or a combination thereof.
Data communication between the processor 202, read only memory 204, random access memory 206, storage interface 208 and the network interface subsystem 210 is suitably accomplished via a bus data transfer mechanism, such as illustrated by the bus 212.
Also in data communication with the bus 212 is a document processor interface 222. The document processor interface 222 suitably provides connection with hardware 232 to perform one or more document processing operations. Such operations include copying accomplished via copy hardware 224, scanning accomplished via scan hardware 226, printing accomplished via print hardware 228, and facsimile communication accomplished via facsimile hardware 230. It is to be appreciated that the controller 200 suitably operates any or all of the aforementioned document processing operations. Systems accomplishing more than one document processing operation are commonly referred to as multifunction peripherals or multifunction devices.
Functionality of the subject system 100 is accomplished on a suitable document processing device, such as the document processing device 104, which includes the controller 200 of
In the preferred embodiment, the engine 302 allows for printing operations, copy operations, facsimile operations and scanning operations. This functionality is frequently associated with multi-function peripherals, which have become a document processing peripheral of choice in the industry. It will be appreciated, however, that the subject controller does not have to have all such capabilities. Controllers are also advantageously employed in dedicated or more limited purposes document processing devices that perform one or more of the document processing operations listed above.
The engine 302 is suitably interfaced to a user interface panel 310, which panel allows for a user or administrator to access functionality controlled by the engine 302. Access is suitably enabled via an interface local to the controller, or remotely via a remote thin or thick client.
The engine 302 is in data communication with the print function 304, facsimile function 306, and scan function 308. These functions facilitate the actual operation of printing, facsimile transmission and reception, and document scanning for use in securing document images for copying or generating electronic versions.
A job queue 312 is suitably in data communication with the print function 304, facsimile function 306, and scan function 308. It will be appreciated that various image forms, such as bit map, page description language or vector format, and the like, are suitably relayed from the scan function 308 for subsequent handling via the job queue 312.
The job queue 312 is also in data communication with network services 314. In a preferred embodiment, job control, status data, or electronic document data is exchanged between the job queue 312 and the network services 314. Thus, suitable interface is provided for network based access to the controller function 300 via client side network services 320, which is any suitable thin or thick client. In the preferred embodiment, the web services access is suitably accomplished via a hypertext transfer protocol, file transfer protocol, uniform data diagram protocol, or any other suitable exchange mechanism. The network services 314 also advantageously supplies data interchange with client side services 320 for communication via FTP, electronic mail, TELNET, or the like. Thus, the controller function 300 facilitates output or receipt of electronic document and user information via various network access mechanisms.
The job queue 312 is also advantageously placed in data communication with an image processor 316. The image processor 316 is suitably a raster image process, page description language interpreter or any suitable mechanism for interchange of an electronic document to a format better suited for interchange with device functions such as print 304, facsimile 306 or scan 308.
Finally, the job queue 312 is in data communication with a parser 318, which parser suitably fimctions to receive print job language files from an external device, such as client device services 322. The client device services 322 suitably include printing, facsimile transmission, or other suitable input of an electronic document for which handling by the controller function 300 is advantageous. The parser 318 functions to interpret a received electronic document file and relay it to the job queue 312 for handling in connection with the afore-described functionality and components.
In operation, image data encoded in a multidimensional color space is first acquired. Histogram data is then calculated in accordance with the acquired image data and a ramp zone associated with the calculated histogram data is detected. A black level associated with the acquired image data is then selectively stretched in accordance with the detected ramp zone so as to generate enhanced image data. The enhanced image data is then output to an associated data storage or an associated display.
According to one example embodiment of the subject application, image data encoded in a multidimensional color space, such as RGB, or the like, is first received by the controller 108 or other suitable component associated with the document processing device 104, by the user device 114, or by any other suitable processing device, as will be appreciated by those skilled in the art. The skilled artisan will further appreciate that while reference is made hereinafter to the controller 108 or other suitable component associated with the document processing device 104 implementing the subject application, other computing devices are equally capable of implementation of the system and method of the subject application. Acquisition of the input image data is capable of occurring via operations of the document processing device 104, e.g. scanning, facsimile, electronic mail, or the like, via an external device, e.g. a digital camera, via a portable storage device (not shown), via communication from a networked device, e.g. the user device 114, or the like. Following receipt of the input image data, a histogram is calculated and normalized by the total number of pixels in the input image, as will be understood by those skilled in the art.
An M-th order backward difference is then calculated from the normalized histogram data so as to generate difference data. In accordance with one embodiment of the subject application, a first order backward difference is applied to the calculated histogram data to generate the difference data. A ramp zone is then calculated in association with the histogram data by the controller 108 or other suitable component associated with the document processing device 104. Data corresponding to a property of the ramp zone is then acquired, e.g. ramp start, ramp stop, ramp length, or the like. The length of the detected ramp zone is then calculated from the zone property data.
A determination is then made whether the ramp start property begins at a predetermined threshold value (Th). When the ramp start has a value above the predetermined threshold value, e.g. ramp start >Th, black stretch is not applied to the input image. When it is determined that the ramp start property has a value below the predetermined value (Th), a second determination is made whether the ramp length exceeds a predetermined threshold value (Th′). In the event that the length of the ramp does not meet the predetermined threshold value, e.g. ramp length <Th′, black stretch is not applied to the input image. When the ramp length is above the predetermined threshold value, a third determination is made whether the histogram count at ramp start is below a predetermined threshold value (Th″). In the event that the histogram count at ramp start is above the predetermined threshold value, e.g. histogram count >Th ″, black stretch is not applied to the input image.
Upon the determinations that the ramp start occurs below the predetermined threshold value (ramp start <Th), that the ramp length exceeds the predetermined threshold value (ramp length >Th′), and that the histogram count at ramp start is below the predetermined threshold value (histogram count <Th″), the input image is tested so as to determine whether or not the input image represents a fog scene image, a partial fog scene image, or a tinted artistic scene image. The classification of the input image as a fog scene, partial fog scene, or tinted artistic scene is capable of being accomplished in accordance with the systems and methods set forth in U.S. patent application Ser. Nos. 11/851,160 and 12/039,225, the entirety of which are incorporated herein. Upon a determination that the input image is a fog scene, a partial fog scene, or a tinted artistic scene, black stretch is not applied to the input image data.
When the input image is determined not to be a fog scene, a partial fog scene, or a tinted artistic scene, an amount of black stretch (Delta) is calculated for application to the input image as a function of the ramp stop property. The calculation of the amount of black stretch (Delta) is discussed in greater detail below with respect to
Turning now to
In accordance with one embodiment of the subject application, a determination is first made whether the input image 502 is in need of black stretch by analyzing the RGB histogram 504 so as to determine whether a “long ramp” towards the black end of the histogram is present. Turning now to
For example, if H is the RGB histogram of bin size 1, define H[i] as the histogram count at the i-th code value, e.g., H[1] is the number of pixels in the image with value 0 in 8-bit code values and H[128] is the number of pixels in the image with value 127 in 8-bit code values, and so on. Therefore, the first order backward difference is D[i]=H[i+1]-H[i].
Turning now to
(ramp stop−ramp start)+1, e.g. ramp length=(18−2)+1, or ramp length of 17.
According to one example embodiment of the subject application, a ground truth is established via the selection of 500 sample images having typical ontology specific to the target application of black stretch, with suitable judgments made on associated image quality, necessary adjustments to improve image quality, amount of adjustments, and the like. It will be understood by those skilled in the art that the determined ground truth enables the identification of those images among the selected sample images that are in need of black stretch. The skilled artisan will therefore appreciate that the derivation of the HT 1312 and LT 1314 values, ramp start 1316 and ramp stop 1318 is suitably based upon the optimization of the rate on detecting images in need of black stretch.
The skilled artisan will appreciate that there are several conditions in which black stretch should never be applied to an input image. A first false positive is illustrated in
Thus, the skilled artisan will appreciate that the forgoing examples illustrate the determination of whether or not to apply black stretch and if so, the amount of black stretch to apply to a received input image. Stated another way, following receipt of an input image, an RGB histogram is calculated and normalized by the total number of pixels. The histograms M-th order backward difference is then calculated. Next, the ramp start, ramp stop and ramp length with respect to ramp zone defined by high threshold value HT and low threshold value LT are calculated. Thereafter, if the “long ramp” (from the histogram) begins at a predetermined starting point, e.g., ramp start <Th; the “long ramp” exceeds a predetermined length, e.g., ramp length >Th′; and the histogram count at ramp start is below a predetermined threshold value, e.g., H[ramp start]<Th″, then the input image is determined to have a legitimate “long ramp”. When the input image does include a legitimate “long ramp” and it is not a fog scene, partial fog scene, or a tinted artistic scene, then black stretch is determined to be applicable to the input image.
The amount of black stretch, Delta, is then calculated as a function of the ramp stop. A Tone Reproduction Curve (TRC) is then calculated that maps (Delta, 255) to (0, 255) to all pixels in the input image. In accordance with one embodiment of the subject application, the TRC is used to build a lookup table, which is then applied to all pixels in the input image.
It will be appreciated by those skilled in the art that the ramp zone is capable of including a dead zone, wherein all code values are 0 in a region between 0 code value and some higher code value, e.g. images from digital cameras, scanners, or the like. Furthermore, at the higher end of the dead zone, a ramp is capable of being found in which the code values increase as per the illustration in
The skilled artisan will note that clipping in this dark region need not necessarily be problematic. For the most part, if the clipping is modest, it enhances the image, and in the case of a noisy image (e.g., caused by a high ISO setting on the digital camera and/or a long exposure), black point clipping is capable of reducing the apparent noise of an image. According to one particular example embodiment of the subject application, the parameters are optimized as follows: M=1, i.e., first order backward difference; HT=0.7E-4, LT=−1.3E-3 for ramp zone; and Th=3, Th′=2, and Th″=0.9E-3.
The skilled artisan will appreciate that the subject system 100 and components described above with respect to
At step 2004, histogram data is calculated in accordance with the acquired image data by the controller 108 or other suitable component associated with the document processing device 104. A ramp zone is then detected at step 2006 associated with the calculated histogram data. The skilled artisan will appreciate that the ramp zone is suitably illustrated in
Referring now to
At step 2104, the controller 108 or other suitable component associated with the document processing device 104 calculates histogram data from the acquired image data. According to one embodiment of the subject application, the histogram data is suitably normalized by the total number of pixels in the acquired input image, as will be understood by those skilled in the art. A first order backward difference is then applied to the calculated l0 histogram data at step 2106 so as to generate difference data. The controller 108 or other suitable component associated with the document processing device 104 then calculates a ramp zone at step 2108 associated with the calculated histogram data. A suitable example of the ramp zone determination is illustrated in
Data corresponding to a property of the detected ramp zone is then acquired by the controller 108 or other suitable component associated with the document processing device 104 at step 2110. In accordance with one embodiment of the subject application, the detected ramp zone property includes, for example and without limitation, ramp start, ramp stop, ramp length, or the like. The length of the detected ramp zone is then calculated at step 2112 from the zone property. As set forth above, in accordance with one embodiment of the subject application, the ramp length is calculated by subtracting the ramp start from the ramp stop and adding one (ramp length=(ramp stop−ramp start)+1).
A determination is then made at step 2114 whether the ramp start value occurs below a predetermined threshold value (Th), i.e. whether ramp start <Th. When it is determined by the controller 108 or other suitable component associated with the document processing device 104 that the ramp start value is greater than the predetermined threshold value Th, black stretching is precluded at step 2132, whereupon operations terminate with respect to
Upon a determination at step 2114 that the ramp start is less than Th, flow proceeds to step 2116. A determination is made at step 2116 whether the calculated ramp length exceeds a predetermined threshold value (Th′). That is, a determination is made whether ramp length >Th′. Upon a determination that the ramp length is less than the predetermined threshold value Th′, flow proceeds to step 2132, whereupon black stretch is precluded from application to the acquired image data and operations with respect to
Upon a determination at step 2118 that the histogram count at ramp start is less than the threshold value Th″, flow proceeds to step 2120. At step 2120, the acquired image data is tested by the controller 108 or other suitable component associated with the document processing device 104. A determination is then made at step 2122 as to whether the acquired image data represents a fog scene image. In the event that the acquired image data is determined to represent a fog scene image or a partial fog scene image, flow proceeds to step 2132, whereupon black stretch is not applied to the acquired image data. Upon a determination at step 2122 that the acquired image data does not correspond to a fog scene image or a partial fog scene image, flow proceeds to step 2124. At step 2124, a determination is made whether the acquired image data corresponds to a tinted artistic scene image. If the acquired image data is determined to represent a tinted artistic scene image, flow proceeds to step 2132, whereupon black stretch is not applied to the acquired image data and operations with respect to
Upon a determination at step 2124 that the acquired image data does not correspond to a tinted artistic scene image, flow proceeds to step 2126, whereupon the amount of black stretch (Delta) to be applied to the acquired image data is calculated as a function of the ramp stop property. In accordance with one example embodiment of the subject application, the calculation of the amount of black stretch is based upon a piece-wise linear fitting function for correlation: amount of black stretch, Delta=0.48*ramp stop+5.2 if ramp stop is between 6 and 30; Delta 0 if ramp stop is less than 6; and Delta=20 if ramp stop is greater than 30. The skilled artisan will appreciate that the preceding formulae are for example purposes only and not intended to limit the subject application thereto.
Once the amount of black stretch (Delta) has been calculated at step 2126, flow proceeds to step 2128. At step 2128, the controller 108 or other suitable component associated with the document processing device 104 applies tone reproduction curve mapping of (Delta, 255) to (0, 255) to all pixels in the acquired image data so as to generate enhanced image data. The enhanced image data is then output at step 2130 to an associated data storage, e.g. the data storage device 110, or an associated display, e.g. the user interface 106. The skilled artisan will appreciate that the methodology of
The foregoing description of a preferred embodiment of the subject application has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the subject application to the precise form disclosed. Obvious modifications or variations are possible in light of the above teachings. The embodiment was chosen and described to provide the best illustration of the principles of the subject application and its practical application to thereby enable one of ordinary skill in the art to use the subject application in various embodiments and with various modifications as are suited to the particular use contemplated. All such modifications and variations are within the scope of the subject application as determined by the appended claims when interpreted in accordance with the breadth to which they are fairly, legally and equitably entitled.
