The present invention relates to the field of calibration. More specifically, the present invention relates to enhancing the calibration of a device invariant to illuminants using colored rays.
A digital still camera (DSC) or video camera (camcorder) has a sensor that is covered by a color filter array (CFA) to create pixel locations. A DSC typically uses red, green and blue (RGB) filters to create their image. Most current camcorders typically use cyan, magenta, yellow and green (CMYG) filters for the same purpose.
A conventional sensor is a charge-coupled device (CCD) or complimentary metal oxide semiconductor (CMOS). An imaging system focuses a scene onto the sensor and electrical signals are generated that correspond to the scene colors that get passed through the colored filters. Electronic circuits amplify and condition these electrical signals for each pixel location and then digitize them. Algorithms in the camera then process these digital signals and perform a number of operations needed to convert the raw digital signals into a pleasing color image that can be shown on a color display or sent to a color printer.
Each color camera has a unique sensor, CFA, and analog electronics system. The sensor and CFA have part-to-part variations. Accordingly, the electronic system needs to be calibrated for each camera. The goal is to make a “real world” scene captured with different cameras look the same when rendered on a display device. In order to calibrate an individual camera, the properties of the individual camera's primary color channels (CMYG for a camcorder; RGB for a DSC) need to be measured so that the individual camera's response to known colors can be quantized.
In the past, to calibrate a DSC, several pictures were taken with the DSC of objects with different colors. The more colors acquired, the better the calibration would be. The data from these pictures would then be transferred to a computer for analysis, and then parameters are adjusted within the DSC based on the analysis so that the DSC produces the best and most realistic colors as viewed by a human observer. As described above, this was necessary because there are part to part differences in each camera such as there are slight variations in each sensor.
Since the captured “real world” scene may be viewed under different illuminating conditions, the adjusted parameters with each DSC need to be illuminant invariant, i.e., they should be able to generate the most realistic colors no matter what viewing condition is involved. In the past, this has been achieved by taking multiple shots of calibration objects under various illuminants and parameter adjustment is carried out for each shot image. The main drawback of these systems is that they are very slow. It takes time to take a number of pictures of different objects under various illuminating conditions. It also takes time to transfer the data to a computer for analysis and calibration because typically the connection is slow.
The color calibration using colored rays method achieves illuminant independence for digital still cameras under multiple illuminating conditions. A constraint ensures similar calibration performance between colored rays set and the Macbeth ColorChecker. The constraint is derived using matrix-vector operations and properties of the Kronecker product. An iterative scheme using orthogonal non-negative matrix factorization with the new constraint is able to obtain the optimal colored rays set for illuminant independent calibration. Then, by acquiring an image of the optimal colored rays set, a camera is able to determine an adjustment matrix for color calibration. Experimental results show that compared to traditional calibration approach for digital still cameras, the colored rays approach gives smaller color error under various evaluation illuminants and eliminates the need of multiple images acquisition.
In one aspect, a method of generating an optimal colored rays set and calibrating devices using the optimal colored rays set comprises deriving a constraint for illuminant independent colored rays, on a first device, applying Orthogonal Non-negative Matrix Factorization (ONMF) with the constraint, on the first device, obtaining the optimal colored rays set using ONMF, on the first device, acquiring an image of the optimal colored rays set, on a second device, minimizing a color error between acquisition and rendition, on the second device and determining an adjustment matrix, on the second device. The first device is offline. Calibrating the devices is in real-time. Calibrating the devices is illuminant independent. The first device and the second device are selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.
In another aspect, a method of calibrating a device comprises acquiring an image of an optimal colored rays set and determining an adjustment matrix for calibrating the device. The method further comprises minimizing a color error between digital still camera acquisition and human visual system rendition. Calibrating the device is in real-time. Calibrating the device is illuminant independent. The device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.
In yet another aspect, a system for generating an optimal colored rays set and calibrating devices using the optimal colored rays set comprises a first device configured for obtaining an optimal colored rays set and a second device configured for determining an adjustment matrix for calibrating the second device based on an acquired image of the optimal colored rays set. The first device is configured for deriving a constraint for illuminant independent colored rays. The first device is configured for applying Orthogonal Non-negative Matrix Factorization (ONMF) with the constraint. The second device is configured for acquiring an image of the optimal colored rays set. The second device is configured for minimizing a color error between acquisition and rendition. The first device is offline. Calibrating the devices is in real-time. Calibrating the devices is illuminant independent. The first device and the second device are selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.
In another aspect, a device comprises a memory for storing an application, the application configured for minimizing a color error between acquisition and rendition and determining an adjustment matrix and a processing component coupled to the memory, the processing component configured for processing the application. The application is configured for calibrating the device in real-time. The application is configured for calibrating the device independent of an illuminant. The device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.
In another aspect, an application stored in a memory of a device, the application comprises an image acquisition component configured for acquiring the raw data for the optimal colored rays set and an adjustment matrix component configured for determining an adjustment matrix by minimizing color error between acquisition and rendition. The device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television and a home entertainment system.
The objective of Digital Still Camera (DSC) color calibration is to find an optimal Adjustment Matrix (AM) which minimizes the perceived color error of a given target set of color patches as shown in
In order to derive the constraint for generating optimal colored rays set in color calibration, it is useful to first introduce the vec(•) operator that transforms a matrix into a vector by stacking the columns of the matrix one underneath the other in sequence. For any matrices, A and B, their Kronecker product is defined as follows:
It is also useful to list some properties of the vec(•) operator and the Kronecker product. For arbitrary matrices T, U, V and W, the following expressions hold:
(T+U)(V+W)=TV+TW+UV+UW (2)
(TU)(VW)=TVUW (3)
(TU)T=TTUT (4)
vec(UVW)=(WTU) vec(V) (5)
In order to achieve illuminant independence, a set of color rays are considered as the input of the color calibration signal flow as shown in
Then, the calibration problem is able to be formulated as
AMMOCSmin(CEMcolor)(CEMcolor2)
where the optimization metric CEMcolor2 is able to be formulated as:
where r is the number of color patches in the input colored rays set. Since
The order of the summations is able to be rearranged and by using the definition of vector 2-norm, the following expression is obtained.
Let w′D=√{square root over (wD)},w′A=√{square root over (wA)},wF=√{square root over (wF)}, and define J as the Jacobian matrix of the nonlinear transformation from CIEXYZ space to CIEL*a*b* space. Then, by using vec(•) operator, the color error metric is able to be formulated as
then the color error metric is able to be stated as
According to the property of Kronecker product in Equation (5), the above equation is able to be expressed as follows:
By this notation,
the color error metric is
CEMcolor2=∥YMOCS·bMOCS−XMOCS∥22 (12)
so bMOCS is able to be solved using least square method as
b
MOCS=(YMOCST·YMOCS)−1(YMOCST·XMOCS) (13)
By substituting the terms in Eq. (13), results in:
Let and apply Equation (4), then
Then, by using Equations (3) and (5):
Then, applying Equation (3) again,
Assuming the spectral reflectance function is sampled in the visible range [400 nm-700 nm] with 5 nm interval, then there are 61 samples for each spectral reflectance and colored ray. So:
By applying Equation (5), the above expression is able to be changed to:
so if the corresponding terms are substituted in Equation (17) by Equations (18) and (20), the resulting expression is:
After applying Equation (3) again on the above equation, the final result is:
The calibration flow using the Macbeth ColorChecker is illustrated in
The mathematical derivation of AM optimization using Macbeth ColorChecker follows the similar pattern as in, U.S. patent application Ser. No. 11/862,107, filed Sep. 26, 2007, entitled, “SYSTEM AND METHOD FOR DETERMINING AN OPTIMAL REFERENCE COLOR CHART,” and for simplicity, the details are omitted here. The final expression for AMMC24 is as follows:
In order to match the calibration performance between MOCS colored rays set C and Macbeth ColorChecker R, the complete condition is to match the corresponding adjustment matrix AM (e.g., bMOCS≈zMC24). Comparing the two expressions (22) and (23), it is noticeable that the two equations are very similar, and the only difference involves the illuminant matrices. Therefore
The original Orthogonal Non-negative Matrix Factorization (ONMF) problem proposed in U.S. patent application Ser. No. 11/862,107, filed Sep. 26, 2007, entitled, “SYSTEM AND METHOD FOR DETERMINING AN OPTIMAL REFERENCE COLOR CHART” is to solve
V
n×m
≈W
n×r
·H
r×m
where W≧0, H≧0
H·H
T
=I (25)
The orthogonality constraint of weight matrix H assures that V·VT≈W·WT.
The concept of colored rays which involves the effect of both spectral reflectance and illuminant has been mentioned. When associating the calibration formation (24) with the original ONMF problem (25), V is defined
then the relationship holds
This is exactly the requirement to match the calibration performance of MOCS colored rays and Macbeth. Therefore, ONMF is able to be easily applied to solve the problem of illuminant independent calibration.
After implementing ONMF on the illuminant weighted set
an optimal
colored rays set C is obtained. The spectral distribution of the colored rays set C is shown in
In order to evaluate the calibration performance of the generated optimal colored rays set C, it was applied to the calibration signal flow as shown in
By using matrix-vector operation and properties of Kronecker product, a new framework to generate optimal colored rays for DSC calibration has been developed. The constraint ensures similar calibration performance between colored rays set and Macbeth ColorChecker. A new optimization scheme using the ONMF method described in U.S. patent application Ser. No. 11/862,107, filed Sep. 26, 2007, entitled, “SYSTEM AND METHOD FOR DETERMINING AN OPTIMAL REFERENCE COLOR CHART,” with the new constraint is developed to obtain the optimal colored rays set for illuminant independent calibration. Experimental results show that compared to traditional calibration approach for DSC, the colored rays approach gives smaller color error under various evaluation illuminants with less run-time requirement.
In some embodiments, the optimal colored rays application 830 includes components/modules to accomplish each task. For example, in some embodiments, a constraint component 832 is utilized to derive a constraint for illuminant independent colored rays. An ONMF component 834 is utilized to apply ONMF with the constraint. A colored rays set component 836 is utilized to obtain an optimal colored rays set after implementing ONMF. In some embodiments, more or less components are included in the color calibration application 830.
In some embodiments, the color calibration application 880 includes components/modules to accomplish each task. For example, in some embodiments, an image acquisition component 882 is utilized to acquire an image of the optimal colored rays. A error minimization component 884 is utilized to minimize color error between digital still camera acquisition and human visual system rendition. An adjustment matrix component 886 is utilized to determine an adjustment matrix which is used to calibrate the computing device. In some embodiments, more or less components are included in the color calibration application 880.
Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system or any other suitable computing device.
To utilize the color calibration using colored rays method, a first computing device obtains an optimal colored rays set by deriving a constraint for illuminant independent colored rays and applying ONMF with the constraint. A second computing device acquires an image of the optimal colored rays. The second computing device then minimizes the color error between digital still camera acquisition and the human visual system rendition. The second computing device ultimately determines an adjustment matrix which is able to be applied, so that the second computing device is calibrated correctly. Once calibrated the computing device is able to acquire images in colors matching the colors seen in real life.
In operation, color calibration using colored rays improves the process of calibrating a computing device by making the process more efficient with the need of only capturing a single image for calibration. A first computing device obtains an optimal colored rays set. A second computing device, to be calibrated, then acquires an image of the optimal colored rays, minimizes a color error and then determines an adjustment matrix for calibrating the computing device. Once the computing device is calibrated, it operates as usual. The utilization of the computing device from the user's perspective is similar or the same as one that uses another form of calibration. For example, the user still simply turns on a digital camera and uses the camera to take pictures. With the camera calibrated properly, the images acquired are the appropriate color.
The present invention has been described in terms of specific embodiments incorporating details to facilitate the understanding of principles of construction and operation of the invention. Such reference herein to specific embodiments and details thereof is not intended to limit the scope of the claims appended hereto. It will be readily apparent to one skilled in the art that other various modifications may be made in the embodiment chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.