Patent Grant
10825132
The present invention relates to image and video processing. More specifically, the present invention relates to systems and methods for transforming images from videos such that the end result videos have a distinct style. The style may be copied or transferred from a reference image.
The interest in image processing within the past decade has been steadily increasing. Video and image processing has developed to the stage where the general public, equipped with a suitably equipped personal computer or workstation, can now perform seeming magic on images and video.
The use of photo and image editing software packages such as Photoshop has allowed manual photo editing to be in the reach of the general public. When it comes to video, there are also corresponding software packages that would allow the general public to edit, splice, and manipulate videos with the same seeming ease that images can be manipulated.
One challenging field, however, is that of automated video or image editing. While there have been numerous successful attempts at automatically manipulating video and animations, these tend to take a large amount of computing power as well as a large amount of time. The public is aware of what computer generated images (CGI) can do when it comes to cinema. And, equally, a sector of the public is aware of how much resources such special effects can take.
When it comes to automated video processing, one task that is still proving challenging is that of style transfer. Style-transfer consists of modifying pictures or videos so that they acquire the artistic style of a different work. Artistic style refers to the brush, strokes, lines, colours, patterns, motifs, and textures that are specific to a given work of art. While manual style transfer can be easy, high quality automated style transfer, especially when it comes to video, is proving to the difficult.
Currently, automated style transfer methods require too much processing power to be performed in real-time. As well, the results of these methods suffer from temporal instability. Instability can best be explained in terms of its opposite, stability. A stable style transfer means that the style elements will stay in place in the video from frame to frame, relative to the objects in the video or animation. A stable style transfer also means that the style elements will not pop up in one frame and then disappear in the next frame.
It should be clear that current style transfer methods cannot be performed in real-time. By this is meant that these methods cannot effect a style transfer at reasonable resolution (e.g. 850×500 pixels or more) at a reasonable framerate (e.g. 30 fps or more) using a single computer with a high-end GPU card.
To highlight this shortcoming, it has been found that initial work by Gatys (see: https://arxiv.org/abs/1508.06576) as used by Johnson (see https://github.com/jcjohnson/neural-style) results in a frame transformation rate of 1 fps (frames per second). These results were obtained even without frame-to-frame instability.
Another prior art effort in this field can be found at https://github.com/genekogan/CubistMirror. This effort implements the base style-transfer algorithm, without stabilizing the style transfer. In this effort, there is nothing to guarantee the temporal coherence of consecutive frames. As such, the style-transfer is unstable and suffers from stylization features jumping around. As well, stylization features can appear/disappear between frames.
Another prior art effort is detailed at https://prisma-ai.com/. In this effort, the style-transfer is effected but mainly for images, with style transfers occurring within about 10 seconds. However, this effort does not offer style transfers for videos or animations in real-time. As well, the effort does not mention stabilization as there is no need for such with single images.
A further prior art effort is detailed at http://www.ostagram.ru/. Again, this effort only offers style-transfer for images. Based on initial work by Gatys, this effort offers significantly better quality, but does not address stability of video and animations stylization. In addition, this effort is quite slow, with processing taking from 1.5 to 2 hours per image.
There is therefore a need for systems and methods that address the shortcomings of the prior art.
The present invention provides systems and methods for use in training a convolutional neural network (CNN) for image and video transformations. The CNN is trained by adding noise to training data set images, transforming both the noisy image and the source image, and then determining the difference between the transformed noisy image and the transformed source image. The CNN is further trained by using an object classifier network and noting the node activation levels within that classifier network when transformed images (from the CNN) are classified. By iteratively adjusting the CNN to minimize a combined loss function that includes the differences between the node activation levels for the transformed references images and when transformed source are classified and the differences between the transformed noisy image and the transformed source image, the artistic style being transferred is maintained in the transformed images.
In one aspect, the present invention provides a method for training a neural network for use in copying a specific visual style to multiple images in a sequence of images, the method comprising:
a) applying noise to a source image from said multiple images to result in a noisy image;
b) sending both of said source image and said noisy image through a convolutional neural network to be trained to result in a transformed image and a transformed noisy image;
c) determining a difference between said transformed image and said transformed noisy image;
d) obtaining a reference style image, said reference style image being an image with a style to be transferred to said transformed image;
e) sending said reference style image through a classifier network, said classifier network being a neural network for object recognition, said classifier network having internal nodes that are activated when an image is sent through said classifier network, wherein an object recognized by said classifier network as being in said image sent through said classifier network is dependent upon a level of activation of said internal nodes as said image passes through said classifier network;
f) determining a level of activation of internal nodes in said classifier network when said reference style image passed through said classifier network to result in a first set of node activation levels;
g) sending said transformed image through said classifier network;
h) determining activation levels of internal nodes in said classifier network when said transformed image passed through said classifier network to result in a second set of node activation levels;
i) comparing said first set of node activation levels with said second set of node activation levels to result in a difference of node activation levels;
j) obtaining said source image from said multiple images;
k) sending said source image through said classifier network;
l) determining activation levels of internal nodes at a specific layer in said classifier network when said source image passed through said classifier network to result in a first set of layer node activation levels;
m) obtaining said transformed image and sending said transformed image through said classifier network;
n) determining activation levels of internal nodes at a specific layer in said classifier network when said transformed image passed through said classifier network to result in a second set of layer node activation levels;
o) comparing said first set of layer node activation levels and said second set of layer node activation levels to result in a difference of layer node activation levels;
p) combining said difference of layer node activation levels, said difference of node activation levels, and said difference between said transformed image and said transformed noisy image to result in a sum of loss functions;
q) adjusting at least one parameter in said convolutional neural network;
r) repeating steps a) to p) and comparing said sum of loss functions with previous sums of loss functions until said sum of loss functions has been minimized.
In another aspect, the present invention provides a method for training a convolutional neural network for transforming images from a sequence of images such that resulting images have a desired style, said method using a training set of training images the method comprising:
a1) obtaining a source image from said training set of training images;
a2) applying noise to said source image to result in a noisy image;
a3) sending said source image through said convolutional neural network to result in a transformed image;
a4) sending said noisy image through said convolutional neural network to result in a transformed noisy image;
a5) determining a difference between said transformed image and said transformed noisy image;
a6) adjusting at least one parameter in said convolutional neural network;
a7) repeating steps a3) to a6) and comparing said difference with previous differences until said difference has been minimized.
In a further aspect, the present invention provides non-transitory computer readable media having stored thereon computer readable and computer executable instructions that, when executed, implements a method for training a neural network for use in copying a specific visual style to multiple images in a sequence of images, the method comprising:
a) applying noise to a source image from said multiple images to result in a noisy image;
b) sending both of said source image and said noisy image through a convolutional neural network to be trained to result in a transformed image and a transformed noisy image;
c) determining a difference between said transformed image and said transformed noisy image;
d) obtaining a reference style image, said reference style image being an image with a style to be transferred to said transformed image;
e) sending said reference style image through a classifier network, said classifier network being a neural network for object recognition, said classifier network having internal nodes that are activated when an image is sent through said classifier network, wherein an object recognized by said classifier network as being in said image sent through said classifier network is dependent upon a level of activation of said internal nodes as said image passes through said classifier network;
f) determining a level of activation of internal nodes in said classifier network when said reference style image passed through said classifier network to result in a first set of node activation levels;
g) sending said transformed image through said classifier network;
h) determining activation levels of internal nodes in said classifier network when said transformed image passed through said classifier network to result in a second set of node activation levels;
i) comparing said first set of node activation levels with said second set of node activation levels to result in a difference of node activation levels;
j) obtaining said source image from said multiple images;
k) sending said source image through said classifier network;
l) determining activation levels of internal nodes at a specific layer in said classifier network when said source image passed through said classifier network to result in a first set of layer node activation levels;
m) obtaining said transformed image and sending said transformed image through said classifier network;
n) determining activation levels of internal nodes at a specific layer in said classifier network when said transformed image passed through said classifier network to result in a second set of layer node activation levels;
o) comparing said first set of layer node activation levels and said second set of layer node activation levels to result in a difference of layer node activation levels;
p) combining said difference of layer node activation levels, said difference of node activation levels, and said difference between said transformed image and said transformed noisy image to result in a sum of loss functions;
q) adjusting at least one parameter in said convolutional neural network;
r) repeating steps a) to p) and comparing said sum of loss functions with previous sums of loss functions until said sum of loss functions has been minimized.
The embodiments of the present invention will now be described by reference to the following figures, in which identical reference numerals in different figures indicate identical elements and in which:
It should be clear that the present invention relates to convolutional neural networks and it should be clear to a person skilled in the art that convolutional neural networks are multi-level, multi-layer software constructs that take in input and produces an output. Each level or layer in the neural network will have one or more nodes and each node may have weights assigned to it. The nodes are activated or not depending on how the neural network is configured. The output of the neural network will depend on how the neural network has been configured, which nodes have been activated by the input, and the weights given to the various nodes. As an example, in the field of image identification, an object classifier convolutional neural network will have, as input, an image and the output will be the class of objects to which the item in the image will belong to. In this example, the classifier “recognizes” the item or items in the image and outputs one or more classes of items to which the object or objects in the image should belong to.
It should also be clear that neural network behaviour will depend on how the neural network is “trained”. Training involves a data training set that is fed into the neural network. Each set of data is fed into the neural network and the output for each set of training data is then assessed for how close it (the output) is to a desired result. As such, in the example, if an image of a dog is fed into a classifier neural network being trained and the output is “furniture” (i.e. the object in the image (the dog) is to be classified as “furniture”), then clearly the classifier neural network needs further training. Once the output of the neural network being trained has been assessed as to closeness (or distance) from a desired result, then the parameters within the neural network are adjusted. The training data set is then, again, sent to the neural network and the output is, again, assessed to determine distance from or closeness to the desired result. The process is iteratively repeated until the output is acceptably close to the desired result. The adjustments and/or parameters of the neural network that produced the result that is acceptable are then saved. A new data training set can then be used for more training so that the output or result is even closer to the desired result.
As can be imagined, depending on the configuration of the neural network, there could be hundreds of levels or layers within the network, with each layer having potentially hundreds of nodes. Since each node may have a weight associated with it, there could be thousands if not millions of potential parameters that can be adjusted during training. The weight associated with each node may be adjusted to emphasize the node's effect or it may be adjusted to de-emphasize that nodes effect or to even negate whatever effect the node may have. Of course, each node may be one in a decision tree towards an outcome or each node may be a step that effects a change on some piece of data (e.g. the input). In the present invention, since the convolutional neural network being trained transforms an image (or a frame from a video) to one that has a specific style, a node may change the parameters of a pixel in the image (e.g. color, contrast, balance, etc.).
As noted above, the present invention seeks to train a convolutional neural network such that the trained neural network would accept an input video's many frames, adjust those frames, and produce output images that, when recombined to form a transformed video, will have a certain visual style that is consistent within the transformed video. Not only that, but the visual style in the transformed video would be stable across its various frames.
With the above in mind,
To train the CNN to produce the desired transformed video, a process similar to that in
In one aspect, the present invention trains the CNN with the help of a classifier network. As noted above, such a classifier neural network outputs the class or category of the objects in an input image with a view towards identifying the object. Referring to
It should be clear that the concept behind this training method is to adjust the CNN until the transformed images it produces are, as much as possible, treated in a similar manner as the reference style image by a classifier network. This would mean that, as an ideal, the classifier network recognizes and decides that the features in the reference style image and in the transformed image are similar and are to be treated and dealt with similarly. This method is designed to ensure that the artistic style from the reference style image is maintained in the transformed images.
In another aspect, the present invention trains the CNN again with the help of a classifier network. Referring to
It should be clear that the concept behind this training method is to adjust the CNN until the source images and the transformed images are recognized and treated by the classifier network as being similar. In one implementation, the activation levels of nodes in layer 3 of the classifier network are noted. For other implementations, the relevant nodes whose activation layers are made note of may be at a different layer. The monitoring of the activation levels of nodes may be for later or higher layers (as opposed to earlier or lower layer) in the classifier network. This method ensures that the features of the source images are maintained, at a high level, within the transformed images.
It should also be clear that while the above description notes the use of a classifier network, other object classification neural network may be used with similar or comparable results. For this implementation, the VGG-16 classifier network (a 16 layer or 16 level neural network) from the Visual Geometry Group from Oxford University was used.
In a further aspect, the present invention trains the CNN to ensure stability across multiple frames in a video. This is done by adding noise to the source images used for training. Referring to
In one implementation, this difference can be the mean square error between the transformed image and the transformed noisy image. The concept behind this method is to train the CNN such that the transformations it applies to the input images are not those that overly amplify changes to a few pixels. This is because a stabilization that produces similar stylization for noisy images should also produce a similar stylization for temporally coherent images (e.g. video frames).
It should be noted that the noise applied to the source image is not simply random noise but noise that has a number of user configurable parameters. These parameters may include:
In one implementation, the noise parameters used were able to produce a CNN that produced stable video stylization suitable for real-time use. In this implementation, the noise count was set at 1000 pixels (for a 512×512 pixel image) and noise level was set to 30 for colors in the range of 0-255.
It should also be noted that while the above description explains each of the three training methods separately, these can be combined into a single implementation. For such an implementation, the source images can be passed through the CNN and the resulting transformed image can be used in the three methods as necessary. The other data and transformed images used by the various methods (e.g. the noisy image, the noisy transformed image, and the sets of activation levels of nodes from the classifier network) can be produced and gathered separately. Once all the data and the comparisons between the data have been performed, the CNN being trained can have its parameters adjusted. Once the CNN parameters have been adjusted, the process can be iteratively repeated until the various loss functions or the various differences noted above have been minimized.
The methods above can be summarized in the flowchart detailed in
In
In
It should be clear that while the various training regimens illustrated in
To clarify one implementation of the present invention,
For feature loss, the differences in late activations between the data set→CNN→classifier chain and the data set→classifier chain are determined. For style loss, the differences between the early activations of the style image→classifier chain and early activations of the data set→CNN→classifier chain are determined. These differences are combined with each other and the mean square error to produce a sum. Once the sum is produced, the parameters in the CNN are then adjusted and the process is repeated until the sum is minimized. The sum therefore acts as a combined loss function whose minimization is sought at every iteration after the CNN parameters have been adjusted.
For clarity, the terms “early activation” and “late activation” are to mean that, in a chain of calculations or computations, the early activations are the values at the beginning or near the beginning of the chain while the late activations are those values at the end of or close to the end of the chain. Thus, assuming values u0, u1, u2 are used to compute v1, v2, v3 and assuming v1, v2, v3 are used to calculate w1, w2, w3 and assuming w1, w2, w3 are used to calculate x1, x2, x3 and assuming x1, x2, x3 are used to calculate y1, y2, y3, and assuming y1, y2, y3 are used to calculate z1, z2, z3 then one can determine late and early activations. Early activations would be the values at the beginning of the chain, i.e. u1, u2, u3, v1, v2, v3, w1, w2, and w3. Similarly, late activations would be the values at the end of the chain, i.e. x1, x2, x3, y1, y2, y3, and z1, z2, z3. Early activations, in this document and in the context of the present invention, can mean the activations of neurons located at the beginning of the flow through the convolutional neural network.
It should be clear that the methods detailed above may be implemented using one or more processors installed on one or more computing devices such as personal computers, servers, and the like. Such devices would include storage systems that included both volatile and non-volatile memory and storage devices.
The embodiments of the invention may be executed by a computer processor or similar device programmed in the manner of method steps, or may be executed by an electronic system which is provided with means for executing these steps. Similarly, an electronic memory means such as computer diskettes, CD-ROMs, Random Access Memory (RAM), Read Only Memory (ROM) or similar computer software storage media known in the art, may be programmed to execute such method steps. As well, electronic signals representing these method steps may also be transmitted via a communication network.
Embodiments of the invention may be implemented in any conventional computer programming language. For example, preferred embodiments may be implemented in a procedural programming language (e.g. “C”) or an object-oriented language (e.g. “C++”, “java”, “PHP”, “PYTHON” or “C #”). Alternative embodiments of the invention may be implemented as pre-programmed hardware elements, other related components, or as a combination of hardware and software components.
Embodiments can be implemented as a computer program product for use with a computer system. Such implementations may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or electrical communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server over a network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention may be implemented as entirely hardware, or entirely software (e.g., a computer program product).
A person understanding this invention may now conceive of alternative structures and embodiments or variations of the above all of which are intended to fall within the scope of the invention as defined in the claims that follow.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8775341 | Commons | Jul 2014 | B1 |
| 10032256 | Anaya | Jul 2018 | B1 |
| 10049297 | Chen | Aug 2018 | B1 |
| 20130054297 | Gagnon | Feb 2013 | A1 |
| 20170256038 | Lee | Sep 2017 | A1 |
| 20170372505 | Bhat | Dec 2017 | A1 |
| 20180068463 | Risser | Mar 2018 | A1 |
| 20180173979 | Fan | Jun 2018 | A1 |
| 20180240257 | Li | Aug 2018 | A1 |
| 20190065908 | Lee | Feb 2019 | A1 |
| 20190095629 | Lee | Mar 2019 | A1 |
| Entry |
|---|
| Google Scholar Search results. |
| Leon, A., et al. “A Neural Algorithm of Artistic Style.” [1402.1128] Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition, Sep. 2, 2015, arxiv.org/abs/1508.06576. |
| “Jcjohnson/Neural-Style.” GitHub, Apr. 15, 2017, github.com/jcjohnson/neural-style. |
| “Deep Learning Helps Transfer Famous Artistic Styles to Videos.” NVIDIA Developer News Center, Oct. 2, 2017, news.developer.nvidia.com/deep-learning-helps-transfer-famous-artistic-styles-to-videos/. |
| “Deep Learning Paints Videos in Style of Art Masters | NVIDIA Blog.” The Official NVIDIA Blog, Jul. 6, 2017, blogs.nvidia.com/blog/2016/05/25/deep-learning-paints-videos/. |
| “Genekogan/CubistMirror.” GitHub, github.com/genekogan/CubistMirror. |
| “Prisma” Prisma, prisma-ai.com/. |
| “Delivering Real-Time AI in the Palm of Your Hand.” Facebook Code, code.facebook.com/posts/196146247499076/delivering-real-time-ai-in-the-palm-of-your-hand/. |
| “Ostagram.” Ostagram, www.ostagram.ru/. |
| Number | Date | Country | |
|---|---|---|---|
| 20190259134 A1 | Aug 2019 | US |
