Patent Application
20180308258
The present principles relate generally to methods and apparatus for learning color palette dictionaries for example-guided image recolorization.
Color transfer is the process to modify the color of an input image according to the color palette of an example image. The color changes could be more or less realistic or artistic depending on the choice of the example image and the final sought appearance. Editing tools for large personal photo collections could be improved with example-based algorithms, especially considering the trend of editing sets of multicontributor images of a given event, such as a party or a wedding. These types of images are often taken with smartphones. Despite the increasingly high quality of such pictures in terms of resolution and imaging devices, these pictures do not reach the high degree of colorfulness of higher end devices.
These and other drawbacks and disadvantages of the prior art are addressed by various described embodiments, which are directed to methods and apparatus for learning palette dictionaries for device-ready example-guided recolorization.
According to one general aspect, a method for determining color palettes of at least one image comprises extracting a color palette comprising a color dictionary for at least one image based on color content in the image. The color palette can be decomposed into a dictionary matrix, an activation matrix, or both. The dictionary matrix is stored for use in recolorization.
According to another general aspect, an apparatus for determining color palettes of at least one image comprises a processor, configured to extract a color palette comprising a color dictionary for at least one image based on color content in the image. The processor is configured to also decompose the color palette into a dictionary matrix, an activation matrix, or both. Recoloring circuitry uses the dictionary matrix for use in recoloring an image.
According to another general aspect, a method for selecting a color palette to recolor an image accesses metadata associated with the image and estimates a scene type of the image using at least one of the metadata and other information. At least one color palette is retrieved corresponding to the scene type. The color palette is used to recolor the image.
According to another general aspect, an apparatus for selecting a color palette to recolor an image comprises means for extracting a color palette comprising a color dictionary. Decomposition means generates a dictionary matrix, an activation matrix, or both. Recoloring circuitry recolors a second image using the dictionary matrix based on a scene type of the second image and the available dictionary matrices.
The details of one or more implementations are set forth in the accompanying drawings and the description below. Even if described in one particular manner, it should be clear that implementations may be configured or embodied in various manners. For example, an implementation may be performed as a method, or embodied as an apparatus, such as, for example, an apparatus configured to perform a set of operations or an apparatus storing instructions for performing a set of operations, or embodied in a signal. Other aspects and features will become apparent from the following detailed description considered in conjunction with the accompanying drawings and the claims.
The present principles may be better understood in accordance with the following exemplary figures, in which:
The described embodiments are directed to methods and apparatus for learning palette dictionaries for device-ready example-guided recolorization. Such recolorization can be used in smartphones so users can edit an image before sharing on a social media outlet.
There are several prior methods for image recolorization.
A first prior art approach uses standard, predefined filters. These predefined color filters are well adapted for smartphones since they are light in terms of memory and computation complexity. However, they suffer from a number of weaknesses. First, they lead to very common looking colors. Second, they only define a global color transformation so the editing capability is limited. This is problematic, for example, if one wants to edit a specific color corresponding to one object.
A second prior art approach uses a dedicated image editing tool, such as Photoshop. These tools enable a practically unlimited array of editing tools. However, they are not adapted for rapid editing and sharing on a smartphone.
A general aspect of the embodiments described herein uses example-guided techniques that benefit from a given example image whose colors are aesthetically desirable. The corresponding palette of the example image is copied to the image to be edited. This approach has many advantages, since one palette, applied to different images will lead to different results, enabling a much larger diversity of looks than predefined color look up tables. In addition, although more computationally demanding than a look up table, such an approach is compatible with embarked processing. However, an example-guided processing system needs an example image as an input. The example image can be provided or manually chosen by the user, which can be a limitation.
One embodiment of such a system learns a set of palettes that can be used for example-guided image editing from a set of desirable example images of a known scene. The example palettes can be used for editing on a device, such as a smartphone, by proposing example palettes possibly corresponding to the current image to be edited. The example scene can be matched to the current picture using metadata, such as GPS (Geostationary Positioning Satellite) information, time of day, and weather, for example. Other information can include the type of scene, such as a sunset, a beach landscape, or a mountain view, or the season, weather, event, or web pictures.
As mentioned, block 110 of
In either case, for a given scene and for a set of m images, a color palette corresponding to each image is extracted by using an extraction algorithm. Every palette is expressed in terms of a known color dictionary of length n. The NBS-ISCC dictionary is one such color dictionary and has 267 entries. The vector obtained from extraction is a vector of proportions that sum to 1. In other words, the pth entry of the vector provides the proportion of the pth color dictionary in the input image. For the NBS-ISCC dictionary, entry 33 of the 267 entries is “brownish pink” and so the 33rd entry in the palette vector of an image describes the proportion of “brownish pink” in that image.
A Non Negative Matrix Factorization (NMF) can be used to extract the dictionary palettes. Other methods can also be used, but the NMF reduces the n×m matrix of palettes to a product of two matrices of positive coefficients:
The matrix W of size n×k is the dictionary matrix, while matrix H is known as the activation matrix and k is the rank of the decomposition, where k<<min(n,m). This can also be seen as a low-rank approximation of the input palette matrix. This low-rank approximation is valid since we assume that, for a set of images of a given scene, the space is not densely populated. In other words, the rank of the input matrix is not n×m.
Another interpretation can be made of the NMF on color palettes. Actually, each column vector of the input matrix (corresponding to a palette vector of an example image) decomposes as a linear combination of example palettes, weighted by the coefficients of the activation matrix. A sub-product of such decomposition is clustering since activation weights can assign each image to a given cluster. Other types of clustering methods can be used that use the coefficients of the activation matrix.
The extraction of palettes from an image or set of images is completely independent from the process of performing Non Negative Matrix Factorization. The extracted palette serves as a color description of each image or set of images. Other method of extracting a color palette can be used, for example, projecting each pixel of the image into a color dictionary, such as the ISCC-NBS System of Color Designation comprising 267 colors.
Each image palette, for example, can be represented as a vector of D entries, each entry being the proportion of a dictionary color in the image, where there are D colors in the dictionary. The NMF guarantees that the coefficients of the decomposition are positive. The activation matrix can be used as input to a clustering method to group images according to their colors. The dictionary matrix can be used to derive palettes used for design and editing.
The Non Negative Matrix Factorization leads to a sparse, compact and descriptive decomposition.
Other methods to find color palettes can be used, such as projecting each pixel into a given color dictionary and using other types of clustering, such as k-means, for example. The method is completely independent on the chosen color dictionary, as well as the clustering method.
Once the lightweight dictionary palettes have been extracted offline, they can easily be embarked on a device such as a smartphone. These dictionary palettes are referred to as lightweight because they have a very low memory footprint.
An example editing scenario can be described by a series of steps. First, a user can take a picture using his smartphone camera. Some metadata can be collected during this acquisition, such as acquisition time, date and location, using a GPS, for example. Additional information can also be retrieved, including online information from a web server, such as the current weather. This additional information can be used to estimate the type of scene being shot. The picture can be processed immediately for recolorization, or stored and available at some other point in time for recolorization. In either case, this picture is referred to as the captured image.
A processor is used to classify the type of scene in the captured image by using some combination of metadata and the additional information. Next, corresponding color palette(s) are retrieved from memory on the smartphone to be used as example palettes. These example palettes that are retrieved can be based on the classification of the picture that was shot, such as the one coming closest to the captured image palette (assuming that the processor has also extracted the captured image's palette). The captured image is enhanced using the selected palette(s) and can then be shared on social networks from the smartphone.
Alternatively, instead of retrieving an example palette that is closest to the captured image's palette, a user can choose the example palettes that give a most desired effect to the captured image, either selected before recolorization of the captured image, or afterward. Recolorization can be performed using the palette chosen in this way.
Allowing an image to be classified by scene type, and either enabling a user to select a sample palette with a desired color effect, or automatically selecting an example palette closest to the image to be recolored, provides significant improvements over prior methods.
This process has particular advantages in that decomposition of example palettes can be done offline and stored on a smartphone using a small amount of memory. Recolorization can be processed online interactively using the stored example palettes. This method also offers a variety of editing capabilities compared to predefined filters, and can be made fully automatic by a user.
The present description illustrates the present principles. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present principles and are thereby included within the present principles.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the present principles and the concepts contributed by the inventor(s) to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.
Moreover, all statements herein reciting principles, aspects, and embodiments of the present principles, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the present principles. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (“DSP”) hardware, read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage.
Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The present principles as defined by such claims reside in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.
Reference in the specification to “one embodiment” or “an embodiment” of the present principles, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.
It is to be appreciated that the use of any of the following “/”, “and/or”, and “at least one of”, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and/or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.
These and other features and advantages of the present principles may be readily ascertained by one of ordinary skill in the pertinent art based on the teachings herein. It is to be understood that the teachings of the present principles may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.
Most preferably, the teachings of the present principles are implemented as a combination of hardware and software. Moreover, the software may be implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPU”), a random access memory (“RAM”), and input/output (“I/O”) interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.
It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or the process function blocks may differ depending upon the manner in which the present principles are programmed. Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present principles.
Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present principles are not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 14306813.8 | Nov 2014 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2015/076244 | 11/10/2015 | WO | 00 |
