The invention relates to an image data processing function for use with or incorporated into a display device and, more particularly, a polynomial based function for allowing control of the colour performance of the display device. Further, the invention relates to a method of configuring the polynomial function for optimal colour performance of the display device according to one or more intended target colour performances.
The problem of variation of display colour characteristics, from different display types, and from device model to device model, and even from display device to display device of the same type due to manufacturing variability is well known. Also well known is the related problem of different image content, for display on the same display device, being created or encoded according to different display colour standards.
The solution to accurately displaying image content created according to a given standard on a non-standard display, or vice versa is to employ a gamut mapping algorithm (GMA) which converts the input image data (usually in R, G, B data format and typically of 8-bit per colour resolution, into modified Rout Gout Bout data which when input to the display device produces the correct colour prescribed for the input RGB data by the given standard. Many such gamut mapping algorithms exist, and the different techniques used have been found to have different strengths and weaknesses in terms of ease of computation and subjective evaluation of the resulting image quality, for different combinations, of source and destination colour gamut and different image content (Computational Color Technology, Henry R. Kang, Chapter 7.9 (SPIE Press, ISBN-10: 0819461199, 15 Jul. 2006), Dugay, F., Farup, I. and Hardeberg, J. Y. (2008)).
Most gamut mapping algorithms involve a conversion from the device dependent colour space (e.g. RGB data) to a device independent colour space (e.g. the CIE XYZ space), followed by a matrix rotation of the colour co-ordinates to fit the destination gamut and reconversion of the result into the colour space of the destination device (U.S. Pat. No. 4,843,573, Tektronix, 1987; US2011/0255101 A1, Eastman Kodak, 2010). The key feature of any GMA is how it deals with colours in the source gamut which are outside the destination device gamut, and therefore cannot be displayed on the device, and regions in the destination gamut which are outside the source gamut and therefore will remain unused if the input source colours are reproduced as faithfully as possible. A straightforward transformation of source colours to equivalent destination colours, and clipping of the output data values to the available range of the destination device will result in only the overlapping area of the two gamuts being utilised, and different input colours resulting in the same output colour as out-of-destination-gamut colours are moved onto the boundary of the device gamut. This problem is illustrated in
In order to prevent the loss of image detail due to data clipping, and retain the full gamut of the destination device to maximise the vividness of output image appearance, GMAs have been developed which perform some linear or nonlinear compression of the input colour data in the device independent colour space to allow input colours to be made more accurate according to the source intention, but retain a degree of relative difference in colours which would otherwise be merged. (Perceptual evaluation of color gamut mapping algorithms. Color Res. Appl., 33: 470-476. doi: 10.1002/col. 20443; US 2011/0255101 A1, Eastman Kodak, 2010; U.S. Pat. No. 6,388,674 B1, Sony, 1999).
One type of perceptually optimised gamut mapping algorithm uses two input data transformation functions, one to accurately map input colours to their source intention point in the destination gamut, and one to retain the full extent of the destination gamut. The output result for a given input colour can then be selected from these two possible outputs, or derived as a blended combination of the two, a function of the input colours proximity to, or probability of being within, a memory colour region (U.S. Pat. No. 6,744,534 B1, Xerox, 2000; U.S. Pat. No. 6,266,165 B1, Lexmark, 1998, US 2010/0194773 A1, Sharp, 2010). The function of this type of method is illustrated in
The disadvantage of these multiple transformation with blending methods is they are computationally more demanding and require more resource in terms of processing power and/or memory requirement. More computationally efficient gamut mapping algorithms have been proposed which are based on a polynomial function only. These have the advantage that they may be optimised for different display types or display-to-display variations simply by reconfiguring the values of the coefficients to each term of the polynomial, and the polynomial process may be embedded in the display control or driver hardware, with only the coefficient values being kept in re-writable storage. U.S. Pat. No. 6,862,029 B1 (Hewlett-Packard, 1999) discloses a method of making automated display characterisation measurements which are then used to calculate coefficients for a polynomial function which provides the desired voltage-transmission curve for a colour display, the polynomial coefficients being stored in the display data channel. However, this publication only describes use of a univariate polynomial function to control the gradation (“gamma”) curve of each colour component separately. No cross-colour dependence or functionality for adjusting colour gamut is described.
U.S. Pat. No. 8,014,028 B2 (Canon, 2008) describes a similar measurement based configuration of a colour transform function to be used in a colour management system. In this disclosure however, although perceptual gamut mapping functionality is provided, the display measurement data is used to provide a detailed transform based on gamut boundary calculations, rather than simply polynomial coefficients, to the colour translation module and, as described, the colour translation module is required to use a look-up table (LUT) to transform some input colours and interpolate the transforms for other colours.
WO 2008/024710 A2 (Texas Instruments, 2006) describes a gamut mapping process based on “rubber-sheet” polynomial calculation. However, the process disclosed requires conversion of the input RGB data to a hue-lightness-chroma colour space, at which point the lightness and chroma data are input to the polynomial to provide colour conversion at constant hue. Also disclosed is the use of a separate LUT function to provide colour appearance preserving conversion of known memory colours, rather than configuring the polynomial function to provide this selectivity.
It is therefore desirable to provide a colour data transformation process that provides the computational simplicity, low data storage requirement and reconfigurability of a polynomial calculation with the degree of functionality provided by more complex perceptually optimised gamut mapping algorithms, without the need for conversion of source RGB data to some intermediate device independent colour space during the data conversion.
According to an aspect, a method for converting image data within a source gamut into modified image data within a destination gamut, is provided including utilizing a processor to operate on input red-green-blue (RGB) data within the source gamut with a multivariate polynomial function to produce modified RGB data within the destination gamut, wherein the polynomial function includes coefficients optimized in accordance with a predefined criteria such that when displayed on a display device having the destination gamut the modified RGB data have a colour appearance representing a perceptually optimised transform of the input RGB data.
According to another aspect, operation of the polynomial function on the input RGB data produces modified RGB data which are accurate in colour appearance for a range of memory colours, display saturated colours (those which are at edges of the source gamut) at corresponding positions at edges of the destination gamut, and display blended intermediate output colours for colours in between the range of memory colours and the saturated colours in the source gamut.
According to another aspect, the predefined criteria is based on calculating perceptually optimised modified target RGB data values in the destination gamut for a set of input RGB data values in the source gamut, and using a numerical optimisation process to minimise a difference between the modified RGB data values achieved by the polynomial function and the modified target RGB data values.
In another aspect, the method includes changing the polynomial function coefficients in accordance with a performance selector input in order to switch colour performance of the display device.
In yet another aspect, the method includes using a 1-dimensional look-up table (LUT) to convert each of the input R, G and B data values in the input RGB data separately to intermediate R, G and B data values, and subsequently inputting the intermediate R, G and B data values to the polynomial function.
According to another aspect, the method includes changing the LUT in accordance with a performance selector input in order to switch colour performance of the display device.
According to still another aspect, a display a display device is provided which includes drive circuitry which carries out a method.
In accordance with another aspect, a computing device is provided which includes at least one of an image rendering program or image data output processor which carries out a method.
According to another aspect, a computer-readable medium is provided having stored thereon a program which, when executed by a processor, carries out a method.
According to another aspect, a method for optimizing coefficients of a multivariate polynomial function with which input RGB data within a source gamut is to be converted into modified RGB data within a destination gamut is provided, including optimizing the coefficients in accordance with a predefined criteria using a processor such that when displayed on a display device having the destination gamut the modified RGB data have a colour appearance representing a perceptually optimised transform of the input RGB data.
In yet another aspect, the predefined criteria includes minimising a difference between the modified RGB data values achieved by the polynomial function and modified target RGB data values.
According to another aspect, the optimising of the coefficients is performed in conjunction with a 1-dimensional look-up table (LUT) to convert each of the input R, G and B data values in the input RGB data separately to intermediate R, G and B data values, and subsequently inputting the intermediate R, G and B data values to the polynomial function.
In accordance with another aspect, values in the LUT and the polynomial coefficients are optimised sequentially in a feedback loop process.
In still another aspect, an optimiser device is provided including a processor programmed to carry out a method.
In another aspect, a non-transitory, computer-readable medium having stored thereon a program which, when executed by a processor, carries out a method is provided.
To the accomplishment of the foregoing and related ends, the invention, then, comprises the features hereinafter fully described and particularly pointed out in the claims. The following description and the annexed drawings set forth in detail certain illustrative embodiments of the invention. These embodiments are indicative, however, of but a few of the various ways in which the principles of the invention may be employed. Other objects, advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the drawings.
a), 6(b), 6(c) and 6(d) represent an example calculation of source data to data for input to a destination device having a colour gamut differing to that of the source.
a), 9(b) and 9(c) are a set of CIE xy diagrams illustrating the performance of an embodiment of this invention using the coefficients of
a) and 12(b) represent a pair of plots showing an example Pre-LUT data set and resulting input data to luminance performance of an embodiment of this invention.
a), 14(b) and 14(c) are a set of CIE xy diagrams illustrating the performance of an embodiment of this invention using the coefficients of
a), 23(b) and 23(c) are examples of the destination device colour performance measurements used by an embodiment of this invention.
a), 24(b) and 24(c) are examples of the source intention colour performance values used by an embodiment of this invention.
In an embodiment of the invention a trivariate 3rd order polynomial function has source RGB data for each pixel in the image as inputs and outputs modified RGB data which results in accurate colour appearance for a range of memory colour, whilst displaying saturated colours (those at the edges of the source gamut) at a corresponding position at the edge of the destination gamut, and resulting in a blended intermediate output colour result for colours in between these two regions in the source gamut.
In a further embodiment, the same polynomial function is reconfigured to provide a luminance boost to mid-brightness saturated colours, while maintaining the accuracy according to the source data intent of unsaturated or memory colours, thereby allowing the destination display to be switched to a “vivid” or “dynamic” mode by a simple switch of the operation polynomial parameters.
In a further embodiment, the trivariate polynomial function of any of the previous embodiments may consist of a reduced number of terms, namely the terms of a second order trivariate polynomial, plus the third order term in the product of R, G and B inputs.
In a further embodiment, as the “vivid” mode will result in the effective data value to luminance (“gamma”) curves for the R, G and B channels individually varying according to the data values applied to the other colour channels, a preliminary LUT conversion is applied to the R, G and B data separately to alter the source gamma curves to some intermediate form between the destination target gamma curve for each colour channel while fully saturated (i.e. input value for the other channels are zero), and the destination target gamma curve for each colour channel while fully unsaturated (i.e. R, G and B input values are all equal).
It is also desirable in an embodiment to provide a means of calculating optimal coefficients for the polynomial functions of the previous embodiments which can account for differences in primary colour position, white point, gamma curve shape of the individual R, G and B channels, and variability of primary colour chromaticity with luminance, in the source device and destination device performance, in order to provide the functionality of the primary and subsequent embodiments. In a still further embodiment of this invention, a means of taking a series of colour measurements of the destination display device and using a genetic algorithm or other numerical optimisation tool to find polynomial coefficients which optimally convert the input RGB data of a set of test colours to pre-calculated target output RGB values for those colours collectively is provided.
In a principal embodiment of this invention conventional R, G, B colour image data is input on a pixel-by-pixel basis to a trivariate 3rd order polynomial function of the type illustrated in
To illustrate this degree of enhanced functionality, it is useful to consider a conventional colour conversion process. The calculation steps of such a process are outlined in
It is clear that a polynomial process consisting of only the terms with coefficients r2 to r4, g2 to g4 and b2 to b4 equal to the elements of the matrix MC would be sufficient to provide the function of the above described conventional transform, if combined with the gamma and inverse gamma functions f1 and f2 and output clipping. However, if the effect of the functions f1 and f2 can be approximated by using some or all of the remaining polynomial terms, possibly in combination with modified values for the coefficients already considered, this may allow an advantageous implantation as the processing for functions f1 and f2 will not need to be implemented. Further advantage may be gained by also using all the polynomial terms in combination to produce the effect of a more complex perceptual mapping for the out-of-gamut colours and a blending effect between a memory colour region and a saturated colour or otherwise defined colour region. In this case, the full functionality of a more complex perceptual gamut mapping algorithm may be achieved using the simplified process flow illustrated in
This shows that the polynomial function is able to account for all the destination display colour inaccuracies listed above, and produce a close match to a perceptually optimised gamut transform calculated output colour value for all colours, without any conversion to an intermediate device independent colour space, or complex matrix based or blending calculation required.
A standard measure of the difference between the ideal target and resulting colour for each sample input may be calculated using the CIE ΔE2000 colour difference measure, and for the example non-standard display the average ΔE2000 value for the difference between the ideal target and the result is improved from 7.98 to 1.29 by the application of the polynomial function, for a test set of over 700 sample input colours.
It is worth noting that the ideal target, and resulting output, CIE 1931 xy positions for some of the sample colours in
In a further embodiment of this invention conventional R, G, B colour image data is input on a pixel-by-pixel basis to a one-dimensional lookup table (LUT) for converting the R, G and B input data separately to an intermediate data level, which is then input to the trivariate 3rd order polynomial function of the type described in the principal embodiment. This process incorporating a “Pre-LUT” conversion is illustrated in
One advantage of the inclusion of the Pre-LUT function is it allows the gamma curve for the R, G and B data channels to be modified prior to input to the polynomial process. One particular use for this modification may be to allow gamma curves of intermediate shape to be produced, when the desired output shape for the gamma curve of a particular channel is different to the desired output shape of the gamma curve for grey input colours (i.e. R, G and B data levels are all equal). This may be the case if the display is required to operate in a mode designed to boost the perceived vividness of images by applying a luminance boost to input colours which are saturated (e.g. mid brightness primary colours), while maintaining the standard luminance of unsaturated colours (e.g. greys).
When the display of this embodiment is required to operate in a mode approximating this type of colour performance, it may be advantageous to use the Pre-LUT function to produce intermediate R, G and B data values having a resulting luminance which is mid-way between the resulting luminance for that data value when the other colour channel has a data value of zero, and the resulting luminance produced by that channel when the other channels have a matching input data value. An example of such a Pre-LUT calculated for the same non-standard display used in the previous example (
a) shows the CIE 1931 xy plot with a set of sample colour positions, with their measured position on the TV in “dynamic mode” (hollow squares), and the resulting position when input to the non-standard display (solid circles). It can be seen that largely due to the significantly smaller gamut area of the non-standard display panel, the majority of the sample colours are displayed with too low a saturation (too close to the centre of the gamut area) relative to the source intention.
These results shows that the polynomial function in conjunction with the Pre-LUT is able to account for all the destination display colour inaccuracies previously listed, and produce a close match to a perceptually optimised gamut transform calculated output colour value for all colours, even when that transform includes a saturation dependent luminance boost for selected colours, without any conversion to an intermediate device independent colour space, or complex matrix based or blending calculation required.
In a still further embodiment, a means of calculating optimised polynomial coefficients is provided. For any given set of sample test colours (RGB data combinations) a corresponding set of modified RGB data values may be found or calculated which, when input to the destination device, result in a colour appearance which for each sample colour is as close as possible to a perceptually optimised transform of the source intention of that sample colour to the destination device gamut. This then provides a set of sample colour RGB combinations and a corresponding set of ideal target RGB combinations. An example set of 50 sample colours and target modified colours, including the Gretag-Macbeth chart colour values, for conversion of sRGB intention input colours for display on the non-standard display of previous examples (the conversion shown in
The process for calculating the target modified RGB data combination for each input sample combination may first calculate the CIE XYZ and/or L*a*b* values for each sample colour according to the source intention. The source intention may be a colour standard such as sRGB in which case the calculation to find intended XYZ values from the RGB data values is well known, or it may be an existing colour performances on a known display, in which case the XYZ values produced by each sample colour on the source display may be measured by displaying the sample colour on the source display and measuring the XYZ and/or L*a*b values with a calibrated colorimeter. The process flow in this case is illustrated in
Having obtained or calculated the source data intention for each of the sample RGB combinations, the process may then search for or calculate target modified RGB data values for each of the input sample colours which when input to the source display produce CIE XYZ or L*a*b* values giving as close as possible a match to the source intention XYZ or L*a*b* values of each sample colour. The modified RGB values may be those that produce the smallest absolute difference in XYZ values, or the minimum ΔE2000 colour difference measure.
These target modified RGB data values may be calculated using a set of colour measurements of the destination device which may then be used to predict the resulting XYZ and/or L*a*b values of other colours, or they may be found using an automated measurement system with a feedback or trial-and-error method to sequentially display a test colour on the destination device and measure the XYZ or l*a*b values which result in order to find the modified RGB values which give the optimal match to the source intention. This potential process is illustrated in
The optimal polynomial coefficients for converting the input sample colour RGB data values into the modified target RGB data values for all the sample colours simultaneously may be produced via a numerical solving algorithm (a numerical solver) implemented by a computer processor and aimed at minimising the sum of differences between the target modified RGB data values and the achieved modified RGB data values produced by the polynomial process. The numerical solver may be aimed at minimising the sum of the square of these differences, or some weighted sum of the differences and their squares, or any other suitable predefined criteria. The numerical solver may be of the evolutionary or genetic algorithm type. The numerical solver may solve for optimum parameters for each colour channel separately (e.g. find optimum values for coefficients r1 to r20, then for g1 to g20, then for b1 to b20), or it may solve them all simultaneously. It may use a calculation based on the CIE XYZ or L*a*b* for each tested modified set of RGB data values achieved by the polynomial to find a colour difference measure, and solve to minimise the colour differences between the target modified RGB data values and the achieved RGB data values.
In an exemplary implementation of this embodiment, the process of which is illustrated in
Included in the input sample colour set (1) are test colours corresponding to those used to produce the destination device RGB to XYZ data set, allowing a corresponding source RGB to XYZ data set (3) to be calculated. An example set of source RGB to XYZ data tables are shown in
X
[a,b,c]
=X
DRa
+X
DGb
+X
DSc Eqn 1a
Y
[a,b,c]
=Y
DRa
+Y
DGb
+Y
DBc Eqn 1b
Z
[a,b,c]
=Z
DRa
+Z
DGb
+Z
DGc Eqn 1c
In the next step (4) the RGB to XYZ data values are “gamma corrected” so as to match the data value to luminance curves of the source intention. As the CIE 1931 Y value corresponds to luminance, this may be achieved with the following calculations applied to all values of n:
These gamma corrected destination RGB to XYZ data values (4) are then used to calculate gamma corrected destination XYZ values for all the RGB combinations in the sample set (5)
The gamma corrected destination RGB to XYZ data values (4) are then used to find the highest luminance value with a CIE xy values within a given difference of the CIE_xy values of the source white point. The CIE xy position of the source white point is given by:
and the CIE xy position of any RGB input on the destination device may be found using equivalent equations for the destination RGB to XYZ data (2). In this example, the highest luminance result for an RGB combination input to the destination device with CIE 1931 xy values matching the source white point xy values to within +/−0.0005 is found. This maximum possible luminance value of the destination display at the source white point is then used to scale source intention sample set XYZ values by multiplying all values in this data set by the ratio of the maximum possible luminance value of the destination display to the source white point luminance. This produces the luminance scaled source intention XYZ sample data set (6).
In the next step (7) the sample set destination gamma corrected XYZ values (5) are blended with the white point luminance scaled sample set source intention XYZ values (6) according to the saturation of each colour in the sample set. This is calculated according to the RGB values of each colour in the sample set as:
Which gives a value between 0 and 1. In order to ensure that a greater selection of colours then only pure grey colours are attributed a saturation value of 0, and therefore are produce a target XYZ value from the blending process matching the source intention XYZ value, this saturation measure may be modified according the equation:
S′=max(0,S*(1+f)−f) Eqn 4b
Where f is a factor between 0 and 1, typically 0.7, which determines the proportion of RGB values which are attributed a saturation of 0, while still maintaining a smooth blend in saturation values from 0 to 1. Each RGB combination in the sample set then has its blended target XYZ value calculated according to the equations:
X
BlendedTarget
=S′×X
GammaCorrectedDestination+(1−s)XLuminanceScaledSource Eqn 5a
Y
BlendedTarget
=S′×Y
GammaCorrectedDestination+(1−s)YLuminanceScaledSource Eqn 5b
Z
BlendedTarget
=S′×Z
GammaCorrectedDestination+(1−s)ZLuminanceScaledSource Eqn 5c
These are used to calculate a blended target XYZ value for each RGB combination in the input sample set (9). These blended target XYZ values may then be converted to L*a*b* values using the well-known conversion to produce the blended target L*a*b* sample set values (10). This conversion needs a reference white XYZ value set, which should be those of the luminance scaled source white point.
These blended target L*a*b* sample set values (10) are then used in the trial-and-error process illustrated in
The target modified RGB data value sample set (13) and the input sample colour set (1) are then used in the polynomial coefficient optimisation process of
The pre-LUTs are then used in conjunction with trial polynomial coefficients to calculate modified RGB data values for every input RGB combination in the sample set (1). These achieved modified RGB values (14) are compared with the target modified RGB values (13) to produce a measure of the accuracy of the combine Pre-LUT and polynomial process in its aim of converting all the RGB combination of the input set into those of the target modified set. This measure may be simply a sum of the absolute differences in each R, G and B value of each combination in the input and target modified sample set, or it may be a square of those differences.
A genetic algorithm which iteratively modifies the set of polynomial coefficients towards the goal of minimising the comparison error measure is used to find optimum coefficients. It can be seen that the specific details described in this exemplary embodiment and illustrated in
This process was used with the non-standard display of previous examples in conjunction with calculated sRGB source intention inputs with to produce the performance shown in
The Pre-LUT and polynomial coefficients may also be further optimised by an iterative process which allows modification of the Pre-LUT values after optimised Polynomial coefficients have been found in conjunction with the initially calculated Pre-LUT values. This may use a similar numerical optimisation process to find modified Pre-LUT values which work in conjunction with the existing optimised polynomial process to produce a better match of the achieved modified RGB sample set to the target modified sample set. Further iterations may be applied subsequently using the numerical optimisation process again to find further optimised polynomial coefficients, to operate in conjunction with the newly modified Pre-LUT values and so on.
Alternatively, to save computational time, the Pre-LUT modification part of this feedback loop could take into account only the pure grey entries (R, G and B values all equal) in the sample set, before again optimising the polynomial coefficients for all entries in the sample set.
Alternatively, the Pre-LUT values themselves could be included as trial parameters with the polynomial coefficients to be optimised simultaneously in a single process.
It may be noted that in the optimised polynomial parameters given for the examples of the non-standard display operating with sRGB source intention in
It also may be noted that the additional image data processing steps required to implement the above described methods in the image data flow of a display device may be assisted by existing image manipulation process in the display device. For example, most LCD display devices use a LUT function in the digital-to-analogue converter (DAC) which converts the input digital image data into analogue voltages for application to each pixel electrode. This conversion is usually configured to take into account the voltage-to-luminance response of the LCD and the intended data value to luminance curve of the expected source data intention. It may however be configured to operate in conjunction with the Pre-LUTs and polynomial processes of this invention to allow further improved colour performance.
The image processor may also be configured to receive an input from a performance selector to implement various user or device control as described herein. For example, the performance selector may be a user input or device interface by which the pre-LUT and/or polynomial coefficients may be changed to switch the colour performance of the display device.
The image processor may represent a stand-alone device, or be incorporated within the source for the input RGB data or the display device itself. For example, the image processor may be included in the driver circuitry of the display device. Alternatively, the image processor function may be included in a computing device as part of its image rendering program or image data output processing.
The non-transitory, computer readable medium storing a computer program which, when executed by a processor, causes the processor to carry out the image processor function described herein represents a further embodiment.
The non-transitory, computer readable medium storing a computer program which, when executed by a processor, causes the processor to carry out the optimiser device function described herein represents a further embodiment.
Although the invention has been shown and described with respect to a certain embodiment or embodiments, equivalent alterations and modifications may occur to others skilled in the art upon the reading and understanding of this specification and the annexed drawings. In particular regard to the various functions performed by the above described elements (components, assemblies, devices, compositions, etc.), the terms (including a reference to a “means”) used to describe such elements are intended to correspond, unless otherwise indicated, to any element which performs the specified function of the described element (i.e., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein exemplary embodiment or embodiments of the invention. In addition, while a particular feature of the invention may have been described above with respect to only one or more of several embodiments, such feature may be combined with one or more other features of the other embodiments, as may be desired and advantageous for any given or particular application.
The invention described herein may be applicable to displays used in mobile devices such as telephones, tablet and portable PCs, televisions and public information displays, wherever the display colour performance may not inherently conform to a given standard, or where the source intention of data input to the display may vary in different usage scenarios, or where the display is required to operate in a variety of colour modes.