Patent Application
20200103283
The present disclosure relates chromatic calibration systems and methods for devices having an embedded camera including determining color calibration data from captured images.
High quality of images captured by embedded cameras, such as those in modern smartphones, has not only become expected, but is a particularly salient selling point in the marketplace. However, the colors represented in image data captured by such embedded cameras are not always faithful to those of the scene or object from which the image data were acquired. This is due to, among other things, limited detector responsiveness to different colors and shortcomings in image processing software on some devices.
Chromatic calibration seeks to measure and adjust the color response of image capture and/or display devices so that colors are represented as close as possible to those of a known color space (e.g., RGB (red, green, blue) CMYK (cyan, magenta, yellow, black) etc.). To calibrate a camera, a known calibration target is photographed and the resulting output from the camera is converted to color values. A correction profile is then built using the difference between the camera result values and the known reference values. Typically, such calibration is performed on each device separately and efforts to relieve the time consuming process of individualized chromatic calibration are ongoing.
An aspect of the invention includes a method of chromatic calibration of a device having an embedded camera that proceeds by illuminating a plurality of color references with at least one light source. Images of the respective illuminated color references are captured with the embedded camera. Color calibration data are determined from the captured images and the color calibration data are stored in association with an identifier of the device.
In an embodiment illuminating includes illuminating the color references with a white light source.
In an embodiment a white light source is a camera flash light on the embedded device.
In an embodiment a determining the color calibration data includes selecting a color model with which a color space is represented in the captured images; determining chromatic values in each of the color channels in the captured images; and subtracting the color channel chromatic value from an ideal chromatic value of the color channel.
In an embodiment the illuminating includes selecting a color of the light source to be of like color as one the color references; and selecting another color of the light source to be of like color as another one of the color references subsequent to capturing the images of the color references under illumination by the previously selected light source.
In an embodiment the determining includes selecting a color model with which a color space is represented in the captured images; determining chromatic values in each of the color channels; and subtracting the color channel chromatic value from an ideal chromatic value of the color channel.
In an embodiment the method may be performed on a device in an enclosure having an aperture formed therein through which the embedded camera is optically aligned and in which the color reference is installed to be optically aligned with the camera.
In another aspect the invention includes an apparatus for chromatic calibration of a an embedded camera that includes an enclosure having an aperture formed therein through which the embedded camera is optically aligned; a plurality of color references each being selectively installable within the enclosure; a processor configured to illuminate the color references with at least one light source; obtain images of the respective illuminated color references captured with the embedded camera; and determine color calibration data from the captured images; and a storage device to store the color calibration data in association with an identifier of the device.
In an embodiment the apparatus includes a processor configured to determine chromatic values in each of color channels in the captured images; and subtract the color channel chromatic value from an ideal chromatic value of the color channel.
In an embodiment the apparatus includes a processor configured to select a color of the light source to be of like color as one the color references; and select another color of the light source to be of like color as another one of the color references subsequent to capturing the images of the color references under illumination by the previously selected light source.
In another aspect the invention includes a computer-readable medium having encoded thereon processor instructions that, when executed by a processor, instructs the processor to illuminate a plurality of color references with at least one light source; obtain images of the respective illuminated color references captured with the embedded camera; determine color calibration data from the captured images; and store the color calibration data in association with an identifier of the device.
The present invention may be described through certain embodiments thereof, which are described in detail herein with reference to the accompanying drawings, wherein like reference numerals refer to like features throughout. It is to be understood that the term invention, when used herein, is intended to connote the inventive concept underlying the embodiments described below and not merely the embodiments themselves. It is to be understood further that the general inventive concept is not limited to the illustrative embodiments described below and the following descriptions should be read in such light.
Additionally, the word exemplary is used herein to mean, “serving as an example, instance or illustration.” Any embodiment of construction, process, design, technique, etc., designated herein as exemplary is not necessarily to be construed as preferred or advantageous over other such embodiments. Particular quality or fitness of the examples indicated herein as exemplary is neither intended nor should be inferred.
The techniques described herein are preferably directed to chromatic calibration of embedded cameras, such as those found in smartphones. Upon review of this disclosure and appreciation of the concepts disclosed herein, the ordinarily skilled artisan will recognize other image capturing applications to which the present inventive concept can be applied. The scope of the present invention is intended to encompass all such alternative implementations.
Enclosure 110 may be constructed to prevent stray light, i.e., light from the outside enclosure 110, from entering its interior. In one embodiment, enclosure 110 is constructed of a black resin by 3D printing operations, although the present invention is not so limited. Enclosure 110 may have an aperture 111 on one side at which a colored reflector 115 may be mounted. Colored reflector 115 serves as a color reference having a specific target color. In certain embodiments, such reflector may be standardized or otherwise known to have a specific reflected light spectrum e.g., narrow spectral band about the desired color. Narrow spectral bands include for example red (590-750 nm, preferably 620-700 nm), yellow (495-590 nm preferably 570-590 nm); blue (380-495 nm, preferably 450-490 nm). Light sources 112 interior to enclosure 110 may illuminate reflector 115. Light sources 112 may also be standardized or otherwise known to have a specific reflected light spectrum. A lens 122 may be mounted to DUT 120 to focus its camera 121 on the illuminated reflector 115 and DUT 120 may capture an image of the illuminated reflector 115 through lens 122. The captured image may be analyzed by computing device 130 to determine calibration data, which may then be stored in database 140 in association with suitable identification information (DUT model, embedded camera model, etc.). Accordingly, any other smartphone or other device of the same model can utilize the calibration data without having to explicitly calibrate the device.
In any preferable embodiment of the invention the camera aperture may be separated from the flash aperture by a distance of at least the radius RL or RF. Is preferable that the flash aperture is separated from the camera aperture by a distance D that is no more than 2×RL, preferably 1.5×RL, more preferably RL. In other embodiments the flash aperture has a radius RF that is smaller than the radius of the camera RL. In preferable embodiments of the invention RF is 0.5, 0.6, 0.7, 0.8 or 0.9×RL. In a still further embodiment of the invention the camera aperture and the flash aperture are mated under a single protective transparent cover such as glass, sapphire or ITO (indium tin oxide). In still further embodiments of the invention one or more of the flash aperture or camera aperture is biased to protrude from the surface of the enclosure. Preferably the flash aperture protrudes from the surface of the enclosure by 0.1, 0.2, 0.3, 0.4 or 0.5×RF.
Returning now to
Computing device 130 may include input/output (IO) circuitry 131, processor circuitry 133, memory circuitry 135 and communications circuitry 137. These resources may realize, among other things typical of a computing device, a lighting control process 172, by which colored light sources are selectively activated and calibration data determination process 176, by which calibration data 166 are determined from captured test image 174. Server device 190 may comprise processor circuitry 191, memory circuitry 193 and communications circuitry 195. These resources may realize, among other things typical of a server device, a database management process 178, by which calibration factors are stored and maintained in database 140.
As illustrated in
In certain embodiments, computing device 130 may be communicatively coupled to light source 112 through a control link 154, such as over a USB connection. For example, control link 154 may carry a suitable signal under control of lighting control process 172 to select a particular light source 112 (e.g., red, green, blue or white) without having to open enclosure 110. In other embodiments, a set of switches (not illustrated) may be mounted on the exterior of enclosure 110 by which the colored light source can be selected.
In certain embodiments, enclosure 110 is constructed, such as by 3D printing, to correspond to dimensional specifications of DUT 120. DUT 120 may be attached to enclosure 110, such as by lens clip 305, and calibration data may be generated by calibration data determination process 176, as described below. Once the calibration data have been generated, they may be conveyed to and stored on database 140. Color correction application 164 executing on any device having the same embedded camera, such as the same model smartphone, may query database management process 197 for the calibration data, which it may then store in its memory 125 as calibration data 166. Color correction application 164 may then apply calibration data 166 to images acquired by camera 121, such as by addition/subtraction of color calibration terms to the respective color channels.
In the discussion that follows, a three letter notation is adopted to distinguish the data: the first letter identifies the light source color, the second letter identifies the reflector color and the third letter identifies a particular color channel. For example, WRB indicates a white light source is incidental upon a red reflector with the blue channel being measured. Subscript “c” refers to values provided by the camera and without subscript refers to ideal values, e.g., WRRc represents white light source, red reflector and red channel for the camera.
When pure white light falls on red, green or blue reflectors, ideally the color contributions in the respective color channel (red, blue and green channels) should be maximum (e.g., red channel value for red reflector should be 255) and the contribution to other color channels should be zero (e.g., green and white channel value for red reflected light should be 0). However, due to suboptimal design and limitations of smartphone camera hardware and software, the value in the respective channel is not 255 and some contributions to other channels are observed (i.e., non-zero values). In certain embodiments, contributions to the respective channel are maximized and the contributions to other channels are minimized or removed. This can be achieved by finding the difference (Δ) between the ideal values and smartphone camera values for each channel. The difference Δ may then be used as a calibration term to correct for chromatic differences between different models and makers of smartphone cameras. These values may be determined according to Table 1 below by, for example, calibration data determining process 176.
The difference values Δ may be stored in database 140 in association with the smartphone and/or camera model number or other distinguishing identifier. A user of any like smartphone may recover the Δ values as calibration data 166 through, for example, color correction application 164. Color correction application 164 may apply color correction data 166 to obtain final values (as indicated by the subscript “f”) for each reflected light configuration according to Table 2 below.
By combining the effects of all three color reflectors, the final calibrated value for each channel may be given by WRf=(WRRc+ΔWRRc)+(WMRc+ΔWGRc)+(WBRc+ΔWBRc) for the red channel, WRf=(WRRc+ΔWRRc)+(WMRc+ΔWGRc)+(WBRc+ΔWBRc) for the green channel and W Bf=(WRBc ΔWRBc)+(WGBc+ΔWGBc)+(WBBc+ΔWBBc) for the blue channel. The color calibration term may be added to/subtracted from the corresponding channel for any arbitrary image. After the operation, the maximum and minimum values may be restricted to 255 and 0, respectively.
The storage areas and memory described herein may be implemented by any quantity of any type of conventional or other memory or storage device, and may be volatile (e.g., RAM, cache, flash, etc.), or non-volatile (e.g., ROM, hard-disk, optical storage, etc.), and include any suitable storage capacity. The storage areas may be, for example, one or more databases implemented on a solid state drive or in a RAM cloud.
Processor circuitry described herein is, for example, one or more data processing devices such as microprocessors, microcontrollers, systems on a chip (SOCs), or other fixed or programmable logic, that executes instructions for process logic stored in memory. The processors may themselves be multi-processors, and have multiple CPUs, multiple cores, multiple dies comprising multiple processors, etc.
Communicative coupling described herein may be, for example, via a networks, which represent any hardware and/or software configured to communicate information via any suitable communications media (e.g., WAN, LAN, Internet, Intranet, wired, wireless, etc.), and may include routers, hubs, switches, gateways, or any other suitable components in any suitable form or arrangement. The various components of the system may include any conventional or other communications devices to communicate over the networks via any conventional or other protocols, and may utilize any type of connection (e.g., wired, wireless, etc.) for access to the network.
System 100 includes a communications circuitry 127, 137, and 195 (e.g., network controller), such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing on communication links 152 and 156. As can be appreciated, the network can be a public network, such as the Internet, or a private network such as an LAN or WAN network, or any combination thereof and can also include PSTN or ISDN sub-networks. The network can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G and 4G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other wireless form of communication that is known.
System 100 may include additional servers, clients, and other devices not shown, and individual components of the system may occur either singly or in multiples, or for example, the functionality of various components may be combined into a single device or split among multiple devices. It is understood that any of the various components of system 100 may be local to one another, or may be remote from and in communication with one or more other components via any suitable means, for example a network such as a WAN, a LAN, Internet, Intranet, mobile wireless, etc.
Color correction application 164 provides an interface such as a graphical user interface (GUI) for a user of the client device (smartphone) to interact with a server. The application may be a database client, etc.
In addition to functionality described above, I/O interface 131 may enable communication between a display device, input device(s), and output device(s), and other components of computing device 130, and may enable communication with these devices in any suitable fashion, e.g., via a wired or wireless connection. The display device may be any suitable display, screen or monitor capable of displaying information to a user of computing device 130, for example the screen of a tablet or the monitor attached to a computer workstation. Input device(s) may include any suitable input device, for example, a keyboard, mouse, trackpad, touch input tablet, touch screen, camera, microphone, remote control, speech synthesizer, or the like. Output device(s) may include any suitable output device, for example, a speaker, headphone, sound output port, or the like. The display device, input device(s) and output device(s) may be separate devices, e.g., a monitor used in conjunction with a microphone and speakers, or may be combined, e.g., a touchscreen that is a display and an input device, or a headset that is both an input (e.g., via the microphone) and output (e.g., via the speakers) device.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a solid state disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, a phase change memory storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
It is to be understood that the software used in embodiments of the present invention may be implemented in any desired computer language and could be developed by one of ordinary skill in the computer arts based on the functional descriptions contained herein. By way of example only, the software may be implemented in the C, C++, C #, Java, Python, Fortran or other programming languages. Further, any references herein of software performing various functions generally refer to computer systems or processors performing those functions under software control.
The computer systems of the present invention embodiments may alternatively be implemented by any type of hardware and/or other processing circuitry. The various functions of the computer systems may be distributed in any manner among any quantity of software modules or units, processing or computer systems and/or circuitry, where the computer or processing systems may be disposed locally or remotely of each other and communicate via any suitable communications medium (e.g., LAN, WAN, Intranet, Internet, hardwire, modem connection, wireless, etc.).
Aspects of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
A processing system suitable for storing and/or executing program code may be implemented by any conventional or other computer or processing systems preferably equipped with a display or monitor, a base (e.g., including the processor, memories and/or internal or external communications devices (e.g., modem, network cards, etc.) and optional input devices (e.g., a keyboard, mouse or other input device)). The system can include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the system to become coupled to other processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, method and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometime be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
The descriptions above are intended to illustrate possible implementations of the present inventive concept and are not restrictive. Many variations, modifications and alternatives will become apparent to the skilled artisan upon review of this disclosure. For example, components equivalent to those shown and described may be substituted therefore, elements and methods individually described may be combined, and elements described as discrete may be distributed across many components. The scope of the invention should therefore be determined not with reference to the description above, but with reference to the appended claims, along with their full range of equivalents.
