The present invention is described in conjunction with the appended figures, wherein:
Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, circuits may be shown in block diagrams in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
Also, it is noted that the embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.
Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine readable medium such as storage medium. A processor(s) may perform the necessary tasks.
An embodiment of the present invention is a process for automating the color grading or color balancing task even when the lighting conditions have dramatically changed, as shown in
Considering now the task of transferring the color from target image (left) to source image (center) in
One aspect of this invention is to notice that if a colorspace transformation (a rotation of the colorspace) is applied to the color data between each iteration of 1D matching, this causes the entire color surface distribution of the center image of
It is an important observation for the purpose of present invention to note that even if a random sequence of rotations is used, the process converges eventually to a valid picture with some successful color transfer properties.
In addition, it was found that the transformation need not be a rotation, although other transformations may be less efficient.
The invention proposes in this example that if 1D slices of the multidimensional color space are matched independently, eventually the entire multidimensional surface will be mapped effectively. To understand this aspect of the invention, it may be beneficial to discuss some background on image histograms.
An image histogram can be understood as the frequency distribution of color in the image. The histogram of a gray scale image is 1D because pixels assume only scalar values of gray—typically from 0 to 255 for instance.
Matching 1D histograms is a well understood process and used in histogram equalization for instance. For example, in case of the two gray scale images shown in
1. Set the histogram of the target image to be stored in some memory array H1 such that the nth element of that array is H1[n] for n=0:255 elements. Similarly set the histogram of source image 2 to be H2[n]. Both distributions are shown in
2. Calculate the cumulative density functions C1, C2 by summing the elements of each memory array. Hence calculate
and
3. Create the required grayscale mapping by scanning the elements of C1 and mapping the grayscale of each entry in C1 to the grayscale value that indexes the value having the least difference in C2
For each m=0:255
Find that value of n which makes |C1[m]-C2[n]| smallest. Define this value as T[m].
The mapping for m is then T[m]
The result is then an array T[m] which contains, for each element m=0:255, a corresponding number that is the grayscale to be assigned. The mapping is shown in
There are variants on this process of course, but the above serves as a mechanism for achieving 1-D histogram matching. The resulting image should then have an intensity distributed similarly to the target image.
Unfortunately, the above method fails for color images because there are more dimensions involved. A naive approach based on the grayscale process above would be to separate each color component (red, green and blue) and match the histograms of each color plane individually. An example of doing this is shown in
However, it is an aspect of the current invention that the above approach can be successful by transforming (step 301) the color space before performing the histogram matching (steps 302 and 303), and then repeating this process iteratively 305, as shown in the diagram of
The central part of the process outlined above is illustrated in greater detail in
Defining this rotation operation as R:
for some angles α,β,γ, then each pixel color vector I(k)=(r, g, b) is transformed in I(k)=(r′, g′, b′) as follows
This operation is can be also referred to as a color space conversion rotating one set of axes in the color space into different set of axes in color space.
In a second step 322, the samples of both distributions are projected onto the new axis by summing in the relevant direction. Hence summing along the transformed red and green color axes yields a projection along the transformed blue axis. This yields three histograms for the three color planes in each sequence. These histograms for the new axes are defined as HI1,HI2,HI3 and HG1,HG2,HG3 for the current and target image, respectively.
Then it is possible using the ID process above to find, in the next step 323, for each axis 1,2,3, the mappings t1, t2, t3 that transform the histograms HI1,HI2,HI3 into HG1,HG2,HG3. The resulting transformation t maps a pixel of color I1, I2,I3 onto t(I1,I2,I3)=(t1(I1),t2(I2),t3(I3)).
The iteration is completed with a rotation in step 324 of the samples by R−1 to return in the original coordinate system.
When no new iterations can modify the iterated samples of I(k), the process is considered to have converged in step 325 to the final solution and the color statistics of I(k) and G are identical.
The process described above is not restricted to 3 dimensions. It can be applied to vector spaces of any dimensionality. The basic steps described above remain the same, the only difference is that the rotation matrices fit the relevant dimension, i.e., for a vector space of 4 dimensions, the rotation matrix becomes a 4×4 matrix etc.
A full process according to an example of the invention includes an initialization step 311 and optional steps 312, 313 to accelerate the process as shown in
INTIALIZATION: Retrieve the pixels of the original image I and the target image G with G containing the desired color distribution. For example, the image pixel I=(I1,I2,I3) encapsulates the red, green and blue components. Set k←0, I(0)←I.
REPEAT:
2. UNTIL convergence on all histograms for every possible rotation
Although the above method can be applied to any kind of mapping, it might be desirable as a first step 312, 313 to use a different technique for registering global transformations between the two distributions. In particular, the novel method is not naturally designed to find rotation between the original and target dataset. This is not to be confused with the rotations used in the PDF transfer process of
A simple solution to this problem is to use the Principal Component Analysis 312 of each of the images.
(I−Ī)t(I−Ī)=UItDIUI (3)
and
(G−
where UI=[uI1,uI2,uI3] and UI=[uG1,uG2,uG3] are the 3×3 orthogonal matrices containing the eigenvectors of the covariance matrices of the pixel color vectors of I and G. The diagonal matrices DI and DG contain the eigenvalues corresponding to the eigenvectors in UI and UG.
Registering the pixel colors vectors of I(0) to the target distribution G is then possible by employing the following linear transformation process 313:
I(0)←(UGDG1/2)(UIDI1/2)−1(I−Ī)+
The color distribution of the image I(0) shares the average value, variance and global color directions with the target image G. This operation cannot handle non-linear transformations but offers an accelerated process if it is used as a first step, i.e., before the actual pdf transfer process.
To obtain a correct transformation, the eigenvectors in UI and UG have to correspond. This is achieved by ordering them with respect to the magnitude of the corresponding eigenvalues and making sure that they do not point in opposite directions. i.e.
∀ i≦3, tuIi·uGi>0 (6)
The color mapping to the original picture transfers correctly the target color palette to the original picture but it might also produce some grain artifacts, as shown in
This effect can be understood by taking the simple example of a linear transformation t of the original picture X: t(X)→aX+b. The overall variance of the resulting picture is changed to var(t(X))=a var(X). Hence a greater stretching (a>1) produces greater noise.
A solution to this noise problem in accordance with an variant of the present invention is to reduce the grain by executing a post-processing process, as shown in
Denoting as I(x,y) the original picture and omitting the coordinates in the expressions to simplify the following discussion, i.e. I, J, ψ, φ, etc. actually refer to I(x,y), J(x,y), ψ(x,y) and φ(x,y), the color transformation is t:I→t(I). The problem posed by the above solution is to find a modified image J of the mapped picture t(I) that minimises on the whole picture range Ω the expression
F=φ·∥∇J−∇I∥
2
+ψ·∥J−t(I)∥2
from which the following can be derived:
φ·J−div(ψ·∇J)=φ·t(I)−div(ψ·∇I) (12)
This is an elliptic partial differential equation. The expression div(Ψ·∇I) can be approximated using standard finite differences as in [17] by: (13)
where Ni corresponds to the four neighboring pixels of i.
This yields the following linear system of step 332:
a(x,y)J(x,y−1)+b(x,y)J(x,y+1)+c(x,y)J(x−1,y)+d(x,y)J(x+1,y)+e(x,y)J(x,y)=f(x,y) (14)
with with Neumann boundary condition ∇J|∂Ω=∇I|∂Ω to match the the gradient of J with the gradient of I at the picture border. The term ∥CJ−CI∥2 forces the image gradient to be preserved. The term ∥J−t(I)∥2 ensures that the colors remain close to the target picture. Without ∥J−t(I)∥2, a solution of expression (7) will be actually the original picture I.
The weight fields φ and ψ affect the importance of both terms of expression (7). Numerous expressions for ψ and φ are possible. For example, the weight field φ can emphasize that only flat areas have to remain flat but that gradient can change at object border, in which case:
(with Φ0 constant). A possible choice 331 for the weight field ψ is to account for the possible stretching of the transformation t. Where ∇t is big, the grain becomes more visible:
where Ψ0 is constant and (∇t)(I) refers to the gradient of t for the color I.
The minimization problem of expression (7) can be solved using the variational principle which states that the integral must satisfy the Euler-Lagrange equation:
The final process is identical to a two-dimensional infinite impulse response (2D IIR) filter in which the coefficients a,b,c,d,e and f only dependent on the original picture I and the mapped picture t(I) and which thus can be computed beforehand. The unknown result picture J can be solved by standard iterative methods 333 like SOR, Gauss-Seidel with multigrid approach. Implementations of these numerical solvers are widely available and one can refer for instance to the respective numerical recipes [14].
The main step of these methods is to solve iteratively for J(x,y) as follows.
(15)
The main computational demands in the first step of the full process are the colorspace conversion and the look up table for mapping pixels. In the second noise smoothing step, the solution of the system requires a substantial amount of spatial processing. It is possible to propose the use of Graphics Hardware from companies such as NVIDIA and ATI to provide a hardware embodiment.
An efficient system can be created by using a general purpose CPU for control and data management while employing the Graphics subsystem for data processing. The main subsystems are proposed as below.
It is well known that Graphics Processing Units (GPU's) can perform multiply/add operations on images [6]. Colorspace conversion is in particular built in as a function on chips from NVIDIA and ATI. The calculation of RI in equation (2) can be performed on such chips directly.
The creation of the 1D mapping look up table can be performed using LUT operations on the GPU. The entire image can be mapped in a single operation using the GPU functionality.
It is well known that all the fundamental operations required for Gauss-Seidel Multigrid optimization can be performed on the GPU [1, 9]. A process description has been previously presented in [5]. That module can be used with minimal change for this application.
It is not required to present all the details of the GPU programming needed to achieve the above, as such programming is considered to be well within the scope of an ordinarily trained person in the relevant field when being taught the steps as described above. This programming implementation will change with generations of GPUs and generations of tool development kits. However, currently, the GPU represents a readily available cheap computational resource that can be taken advantage of for the purpose of implementation the present invention. With the processes as outlined above, frame rates at near real time are achieved with current hardware.
All embodiments of the present invention conveniently may be implemented using a conventional general purpose computer or micro-processor programmed according to the teachings of the present invention, as will be apparent to those skilled in the computer art. Appropriate software may readily be prepared by programmers of ordinary skill based on the teachings of the present disclosure, as will be apparent to those skilled in the software art.
A computer may implement the methods of the present invention, wherein the computer housing houses a motherboard which contains a CPU, memory (e.g., DRAM, ROM, EPROM, EEPROM, SRAM, SDRAM, and Flash RAM), and other optional special purpose logic devices (e.g., ASICS) or configurable logic devices (e.g., GAL and reprogrammable FPGA). The computer also includes plural input devices, (e.g., keyboard and mouse), and a display card for controlling a monitor. Additionally, the computer may include a floppy disk drive; other removable media devices (e.g. compact disc, tape, and removable magneto-optical media); and a hard disk or other fixed high density media drives, connected using an appropriate device bus (e.g., a SCSI bus, an Enhanced IDE bus, or an Ultra DMA bus). The computer may also include a compact disc reader, a compact disc reader/writer unit, or a compact disc jukebox, which may be connected to the same device bus or to another device bus. Examples of computer readable media associated with the present invention include compact discs, hard disks, floppy disks, tape, magneto-optical disks, PROMs (e.g., EPROM, EEPROM, Flash EPROM), DRAM, SRAM, SDRAM, etc. Stored on any one or on a combination of these computer readable media, the present invention includes software for controlling both the hardware of the computer and for enabling the computer to interact with a human user. Such software may include, but is not limited to, device drivers, operating systems and user applications, such as development tools. Computer program products of the present invention include any computer readable medium which stores computer program instructions (e.g., computer code devices) which when executed by a computer causes the computer to perform the method of the present invention. The computer code devices of the present invention may be any interpretable or executable code mechanism, including but not limited to, scripts, interpreters, dynamic link libraries, Java classes, and complete executable programs. Moreover, parts of the processing of the present invention may be distributed (e.g., between (1) multiple CPUs or (2) at least one CPU and at least one configurable logic device) for better performance, reliability, and/or cost.
While detailed descriptions of one or more embodiments of the invention have been given above, various alternatives, modifications, and equivalents will be apparent to those skilled in the art without varying from the spirit of the invention. Moreover, except where clearly inappropriate or otherwise expressly noted, it should be assumed that the features, devices and/or components of different embodiments can be substituted and/or combined. Thus, the above description should not be taken as limiting the scope of the invention, which is defined by the appended claims.