Patent Application
20210134014
The present disclosure relates to a method and device for color similarity evaluation, namely comparing two colors, applicable to areas such as digital image processing or computer vision.
Color differences are of great importance in many applications, particularly when image processing or computer vision techniques are to be employed. Examples of such application topics are mineral identification, dental ceramics, wine color analysis, corrosion detection and, in a more generic field, image segmentation. The latter is widely used and often a critical step in image and video processing algorithms. The noise in the outcome of this step affects the results of the following processing, namely in segmentation, detection or appearance matching, possibly preventing a proper operation. Despite the many years of research, image segmentation is still an unresolved research topic, with color identification playing a key role. The CIEDE2000 [3] metric was recommended by CIE in 2000 for industrial color difference evaluation. However, its benefits have been impaired by the computational cost.
The CIEDE2000 [3] formulation has been as an international standard since 2014, which hereby is incorporated by reference in its entirety and in particular in its calculation of color differences, with reference ISO CIE 11664-6:2014 (CIE S 014-6/E:2013), and has a strong focus on industry applications. This metric has been receiving increased attention, showing high potential in discriminating between colors, especially those with higher similarity. Using the CIEDE2000 [3] metric (ΔE00) allows to quantify how humans perceive the difference between two colors. Nevertheless, its wider usage has been impaired due to the high computation cost, which prevents its application in many scenarios, particularly those with real time constraints.
Even though the technological evolution has enabled higher computation capabilities, there is a large number of scenarios that remain unfeasible due to the high quantities of visual information or restricted hardware.
Color based algorithms can be severely affected by illumination and the discrimination between closer colors is rarely easy. The CIEDE2000 metric has been increasingly used in many applications, enabling a coarser or finer discrimination based on the metric threshold that is selected. The ΔE00 color difference formula was designed so as to characterize the perceptual color difference that humans have and can be summarized by equations 1 to 28:
where KL, KC and KH are correction factors related to Lightness, Chroma and Hue, respectively. In our experiments these values were set to 1 as defined in the CIE standard. The SL, SC and SH parameters are weighting factors; ΔL′, ΔC′ and ΔH′ are the Lightness, Chroma and Hue differences. L1, a1, b1 and L2, a2, b2 are two colors in the CIELAB color space, with ISO 11664-4:2008 (CIE S 014-4/E:2007).
The edge detection methodology presented in [4] combined edge maps from different sources in order to upscale depth information with limited spatial resolution. Some alternatives were shown and the one that provided the best results was comprised of a combination of edge maps for intensity and color difference. The color difference edge map was obtained by usage of the CIEDE2000 color difference, in order to obtain a perceptual color edge detection that follows the characteristics of the human visual system. Although this solution provided the best results, it had the cost of a high computational complexity. For color similarity decisions using ΔE00, values equal or bellow 3 are typically applied; for values above 10, two colors are considered to be very different.
The CIEDE2000 [3] metric has other concurrent metrics, namely other metrics proposed by CIE like, CIE76 (also know as DeltaE CIELAB), CIE94 and DeltaE CMC. Nevertheless, several studies have concluded that more robust results can be obtained by employing ΔE00 even with the default parameters. Better results may be obtained by tuning formula factors to a given application characteristics.
These facts are disclosed in order to illustrate the technical problem addressed by the present disclosure.
The important benefits of the ΔE00 metric are often impaired by its complexity. By analyzing the ΔE00 formula (presented above and described in detail in [3]), the high complexity of several of the terms is noticeable, resulting in a formula with a large computational cost. Hence, the use of this technique in many application scenarios is often unfeasible, particularly when the computational resources are more limited. Moreover, applications that need to perform multiple ΔE00 calculations will be seriously affected with high computation times. As an example, consider two images with resolutions 720×480 and 1920×1080 pixels; in the former one will have to process 0.3×106 pixels and in the latter 2.1×106, resulting in much more calculations. The drawback of the computing cost of ΔE00 becomes even more evident when considering video—with several images per second—and the increasingly penetrating 4K and 8K image resolutions. To overcome this problem and widen the applicability of this formula and its benefits to more scenarios, it is disclosed a method and device for a significantly faster decision on color similarity, while retaining the ΔE00 benefits and capabilities.
To decrease the computation times, the disclosed method and device apply a set of filtering modules where simpler calculations are performed and an assessment is made: (1) the two input colors are different and no further calculations are needed; (2) a decision can not be made with certainty and further analysis is required. Generically, when comparing a pair of colors, these will be an input to the modules and ΔE 00 will only be fully computed for pairs that may have higher perceptual similarity and did not had a decision made by the lighter weight modules. Hence, the number of times the full ΔE 00 metric is calculated, and consequently the total number of operations required, is effectively reduced, resulting in lower processing times, while maintaining the same accuracy. This is supported by the fact that the expected outcome is not a precise distance measure, but a decision on the similarity or not of a pair of colors. Nevertheless, it is still necessary to identify and establish relations between the filtering modules and ΔE 00 so that the results are equal to the ones obtained by only applying ΔE 00, thus maintaining the metric benefits.
Colors are often represented in the RGB color space. This is particularly true when dealing with images, which are typically acquired in the RGB color space. Since the ΔE00 metric makes use of the CIELAB color space, it often requires an additional color conversion step, resulting in extra calculations. It is advantageous for the filtering modules to avoid color space conversions as much as possible, more specifically, that they operate on the RGB color space. This space can be represented as a cube; hence, the difference between two colors can be seen as a difference vector inside the space. With this insight, disclosed method and device put forward three properties to be explored: the module of the difference vector between two colors; the angle between two color vectors; the lightness dissimilarity of the colors. Unlike the first two, the latter is not computed in the RGB space, requiring a color conversion (for example, converting to CIELAB lightness using CIELAB-CIEXYZ conversion and system values from standard ITU-R BT.709-6, as detailed below). Throughout the remainder of this document these properties will be referred to as Module, Angle and Lightness, respectively.
Given two colors in RGB, C1=r1, g1, b1
and C2=
r2, g2, b2
, the Angle, Module and Lightness properties are calculated as follows:
ΔL is given by the following equations,
where C′1 and C′2 the normalized color vectors.
The calculations on equations 32 and 33 are performed on each component of the vectors and are here presented in a vectorized form. The referred properties have well defined limits: the angle between two color vectors (in the RGB color space) is limited between 0 and 90 degrees; the difference vector is limited by 0 as lower limit, if the vectors are equal, and 441.673 if they are on opposing vertices of the cube; the difference of lightness is limited by 0 and 100. The usage of these properties is justified by: (1) their simplicity, especially when compared to ΔE00 as can be observed in equations 1 to 28; (2) their well defined limits.
If a certain maximum threshold of ΔE00 is set, i.e., if we define a value of ΔE00 above which two colors are considered to be perceptually different, then it is possible to reduce the number of calculations made for colors that are expected to give a color difference higher than the defined limit, by thresholding the Angle, Module and Lightness. Therefore, it is necessary to identify relations of Angle, Module and Lightness and the different outcomes of the ΔE00 metric. To understand such relations and the variability of the many parameters of ΔE00, a set of approximately 5 million colors were used and differences calculated using data from the images present in the Caltech 256 [1] dataset as basis pixels.
It is disclosed a method for color similarity evaluation between a pair of color, said method comprising the steps of:
In an embodiment, the one or more filters comprise a module filter wherein the calculation of said filter comprises calculating the module of the difference vector between the first and second colors represented as 3D coordinates in the RGB color space.
In an embodiment, the one or more filters comprise an angle filter wherein the calculation of said filter comprises calculating the angle between the two vectors of the first and second colors represented as 3D coordinates in the RGB color space.
In an embodiment, the one or more filters comprise a lightness filter wherein the calculation of said filter comprises calculating the lightness difference between the first and second colors, in particular the lightness difference between the first and second colors in the CIELAB color space.
In an embodiment, the one or more filters comprise a module filter and an angle filter.
In an embodiment, the one or more filters comprise a module filter and an angle filter, calculated in succession, thus calculating the angle filter after calculating the module filter.
In an embodiment, the one or more filters comprise a module filter, an angle filter and a lightness filter, calculated in succession, thus calculating the lightness filter after the angle filter, and calculating the angle filter after calculating the module filter.
In an embodiment, the one or more filters comprise a module filter, an angle filter and a lightness filter, calculated in parallel.
In an embodiment, the one or more filters comprise a module filter, an angle filter and a lightness filter, calculated in parallel and the calculation of the one or more filters is interrupted when at least one of the calculated filters is above a threshold predetermined for said one filter.
In an embodiment, the threshold predetermined for each filter is pre-obtained from the ΔE00 threshold.
In an embodiment, the threshold predetermined for the lightness filter is pre-obtained from the ΔE00 threshold and a using a linear regression.
In an embodiment, the threshold predetermined for the module filter and/or the threshold predetermined for the angle filter is pre-obtained from the ΔE00 threshold using a machine learning algorithm, in particular using a random forest algorithm.
It is also disclosed a non-transitory storage media including program instructions for implementing a color similarity evaluation device, the program instructions including instructions executable to carry out any of the disclosed methods.
It is also disclosed a color similarity evaluation device comprising said non-transitory storage media according and a data processor arranged for executing said instructions.
It is also disclosed a color similarity evaluation device arranged to carry out the steps of any of the disclosed methods.
The following figures provide preferred embodiments for illustrating the description and should not be seen as limiting the scope of invention.
The thresholds indicated in
This regularization was introduced in the training data as to shift the predicted values so that the possible errors do not influence the final decision. While this translates into predicted thresholds different from the optimal values, it is guaranteed that they are always above the corresponding optimal values and thus always applicable. This translates into a performance deterioration, while assuring corrected predictions. Comparison of predicted thresholds using regularized and non-regularized data are shown in
With the derived models it is possible to obtain, for any ΔE00 value, thresholds for Module, Angle and Lightness so that any two colors whose difference surpasses those thresholds can be classified as having a ΔE00 color difference higher than the specified value. This is very advantageous as it can be mapped into a structure that efficiently cuts calculations, resulting in a significant performance gain without jeopardizing ΔE00 properties. Each threshold can be used to reduce the amount of ΔE00 calculations necessary for a given application that sets a limit for ΔE00. However, the color combinations that each threshold cuts are not the same.
Based on the concepts and relations described, different strategies can be conceived towards a faster decision of color similarity based on the ΔE00 metric. In this section, two embodiments are put forward and discussed: (1) filtering models deployed in cascade (i.e. calculated in succession); (2) filtering models deployed in parallel. Moreover, it is discussed an embodiment applicable to large arrays of color differences to be computed, namely in image processing, where the initial proposals for color difference calculation can be used, specifically taking advantage of the widely spread multi-core hardware.
The first proposed strategy for efficient ΔE00-based color decision consists of deploying the filtering modules in a cascade (i.e. calculated in succession), as illustrated in
Each module of the cascade is intended to make a decision: if colors are different then a cascade output is made; or, uncertainty exists and processing is passed to the next module of the cascade. Hence, we can expect that it is desirable to have faster modules, with smaller computation weight, executed first, preferably pruning as much pixels as possible. Nevertheless, determining the most adequate cascade structure, i.e., to identify the order of the operations, is not necessarily defined by speed of execution. It is important to note that the cascade structure may be dependent on the hardware architecture. Moreover, the value of the ΔE00 threshold to be considered may influence the best cascade structure. Hence, optimized embodiments are described further below.
The cascade approach aims to sequentially perform operations that prune the number of color pairs requiring the calculation of ΔE00. As initial blocks of the cascade are unable to decide on color dissimilarity and ultimately ΔE00 must be calculated, the total number of calculations would in fact be greater than just for the ΔE00 calculation. Hence, the benefits of the cascade strategy derive from a greater number of colors being filtered by earlier blocks.
An alternative strategy to the cascade is proposed, were the different blocks are executed in parallel, as illustrated by the block diagram of
While this approach is sound, its implementation is not as straightforward as the cascade strategy. In fact, several implementation details must be taken into account under the penalty of jeopardizing possible gains.
The strategies described are generic for a decision on color similarity. As previously stated, the benefits of the parallel strategy are more evident when multiple core CPUs can be used and the processing distributed.
When a large array of colors, for example an image, is to be processed, i.e., compared to one or more reference colors, this data can be sequentially processed. In this case, each color would be individually and sequentially passed as input along with the reference color and the desired threshold. In other words, one pair of colors at a time. This is illustrated in
The order of the blocks in the cascade strategy (i.e. the order in which they are calculated in succession) influences the outcome of the overall process, more specifically the time reduction that can be obtained when compared to the calculation of the ΔE00 metric. The ϕE00 block will be the last of the cascade, but it is still necessary to identify what the remaining blocks should be and their order. Given the three properties put forward, there are 15 possible cascade structures as identified in Table 1. Each of these possibilities performs different pruning on the color differences. Moreover, the complexity of the cascade will depend on the restrictions imposed, namely the required ΔE00 decision threshold.
To evaluate the cascade possibilities we implemented a simple segmentation algorithm that, given an image, a color and a ΔE00 threshold, produces a new image that highlights regions where the colors are bellow the specified threshold. Each structure was assessed using a bank of generic colors as specified in [5] and images from the Caltech 256 dataset [1]. A subset of 4K resolution images from the RAISE dataset [6] and a small set of 8K computer generated images were also used. Each instance of the color bank was compared with the color of each pixel of the images of the three subsets. Moreover, the evaluation was performed in three different computer architectures to understand the impact over the cascade structure. Specifically, the following architectures were used: (1) a desktop computer equipped with an Intel i7 4471 processor @ 3.5 Ghz ; (2) a Raspberry Pi 2 Model B; (3) a Samsung Galaxy S6 smartphone. The algorithms were implemented in C++ for the first two environments and in Java for the smartphone. In these experiments no explicit GPU acceleration was used and the CPU time and cycles allocated the process were measured.
The results presented in Table 2 clearly show the advantage of the cascade approach in alternative to the full computation of the ΔE00 formula for a dekstop computer. Independently of the filtering blocks—Module, Angle or Luminance—their order and the underlying ΔE00 threshold, the cascade strategy always achieved significant gains in performance. This is true when measuring processing time as well as CPU cycles. Since there is a clear relation between the number of CPU cycles and the overall processing time, in the following results CPU cycles will only be presented when justified to convey additional information.
The increase of the ΔE00 threshold implies some changes in the ranking and the overall gain decreases. Nevertheless, the time reductions enabled by the cascade strategy remain very significant (above 70%) and the results show a consistency of the cascade structure. Even though there are some changes in the ranking, namely LMA-LAM and M-ML, the differences in the relative gain are almost negligible. The results show that overall greater gains are achieved when Module is the first bock of the cascade. This is consistent with the fact that: (1) it is a lightweight operation; (2) it does not require a color conversion; (3) it enables the filtering of a large number of color differences as can be observed in
The results of the experiments on a Raspberry Pi 2 Model B are consistent with the results obtained in a desktop computer, with relative gains of the same order, despite the more limited resources. Also, as with the desktop computer, there are some modification in the ranking of the cascade structure that are due to almost negligible differences in the relative gain. It is noteworthy the fact that, as in the desktop case, Module has a high impact and for ΔEoo=3 it provides the best result by itself.
The rankings for Android present some differences with regards to the previous cases, especially with desktop, with the ranking starting with MA being followed by AM, M or AML, or starting with AML and being followed by MAL, AM, etc. Nevertheless, the gains are very significant and only slightly lower (between 65% and 89%) than in the previous cases. To this undoubtedly contributes the differences in the computer architectures, but also the requirements imposed on the implementation languages. For desktop and Raspberry the blocks were implemented in C++ and for Android, Java was used.
The proposed embodiments enabled very significant gains in all the hardware platforms used. Slightly lower gains were obtain when using Android, to which are not agnostic not only the platform differences, but also the implementation language itself. Nevertheless, the differences in the relative gain are minimal. Even though the gains have a dependance on the desired decision boundary on the result of ΔE00, they are very high for useful values of the metric; larger values of ΔE00 will have decreasingly lower entropy with values greater than 7 considered to represent two very different colors.
The best structure for the cascade strategy presented slight variations, but overall a small set of possibilities can be identified, to be used horizontally with very small decreases in the relative gain. Specifically, for execution on a desktop computer, it is possible to identify the structure comprised of the filtering blocks Module-Angle followed by the ΔE00 calculation block as providing the best results or just slightly bellow. These same conclusions can be applied to the Raspberry Pi and Smartphone cases. As previously, the structure comprised of the filtering blocks Module-Angle followed by the ΔE00 block can be identified as a good compromise for a structure that can be applied independently of the range of target ΔE00 values. For these two cases, the identified cascade structure does not only provide the best result more often, but the differences are nevertheless very small.
While values of ΔE00 above 10 are not considered, it should be noted that colors with ΔE00 distance above this value are considered to be notably different. Nevertheless, the knowledge resulting from the reported analysis and proposed models (see
The filtering properties analysed—Module, Angle and Lightness—enabled very high processing gains using different strategies and hardware platforms. Nevertheless, other properties can be analysed and their benefits studied. Moreover, it is possible to use a hybrid strategy combining the cascade and parallel strategy.
The term “comprising” whenever used in this document is intended to indicate the presence of stated features, integers, steps, components, but not to preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. Flow diagrams of particular embodiments of the presently disclosed methods are depicted in figures. The flow diagrams do not depict any particular means, rather the flow diagrams illustrate the functional information one of ordinary skill in the art requires to perform said methods required in accordance with the present disclosure. It will be appreciated by those of ordinary skill in the art that unless otherwise indicated herein, the particular sequence of steps described is illustrative only and can be varied without departing from the disclosure. Thus, unless otherwise stated the steps described are so unordered meaning that, when possible, the steps can be performed in any convenient or desirable order.
It is to be appreciated that certain embodiments of the disclosure as described herein may be incorporated as code (e.g., a software algorithm or program) residing in firmware and/or on computer useable medium having control logic for enabling execution on a computer system having a computer processor, such as any of the servers described herein. Such a computer system typically includes memory storage configured to provide output from execution of the code which configures a processor in accordance with the execution. The code can be arranged as firmware or software, and can be organized as a set of modules, including the various modules and algorithms described herein, such as discrete code modules, function calls, procedure calls or objects in an object-oriented programming environment. If implemented using modules, the code can comprise a single module or a plurality of modules that operate in cooperation with one another to configure the machine in which it is executed to perform the associated functions, as described herein.
The disclosure should not be seen in any way restricted to the embodiments described and a person with ordinary skill in the art will foresee many possibilities to modifications thereof. The above described embodiments are combinable. The following claims further set out particular embodiments of the disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 109847 | Jan 2017 | PT | national |
| 17151878.0 | Jan 2017 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/IB2018/050184 | 1/11/2018 | WO | 00 |
