The present application is based on, and claims priority from, Korean Patent Application Number 10-2017-0054474, filed Apr. 27, 2017, the disclosure of which is incorporated by reference herein in its entirety.
The present disclosure in some embodiments relates to a generative adversarial networks.
The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.
Manners of interconnections from concepts to concepts, from objects to objects, and from persons to persons, etc. are ubiquitous cross-domain relations. Cross-domain relations are, in many cases, natural things happening to people. For example, we are aware of the relationship between English texts and their French translation and vice versa. In addition, we select the jacket of suit to wear in style coherent to pants and shoes.
The computer′ problem of whether it can obtain a humanlike capacity to associate domains of two different kinds of images may be reconstructed with a conditional image generation problem. In other words, finding a mapping function from one domain to another is analogous to being responsive to given images in one domain for generating corresponding images in the other domain. Conventionally, such image-to-image translation has been studied mainly by methods in which a training set composed of mutually paired images is used for learning the mapping between the input images and the output images. However, it is very troublesome to prepare paired training data, and the data are unusable in many tasks. The recent introduction of Generative Adversarial Networks (GAN) has led active attempts to apply the algorithm to conditional image generation.
The present disclosure in some embodiments provides a new GAN-based architecture for learning cross-domain relations between two sets of unlabeled images.
In accordance with some embodiments, the present disclosure provides a generative adversarial networks-based or GAN-based architecture in which a first GAN (generative adversarial networks) learns a translation of images in domain A to images in domain B, and a second GAN learns a translation of the images in domain B to the images in domain A. A closed loop formed by the first GAN and the second GAN causes sample images to be reconstructed into an original domain after being translated into an target domain. Therefore, loss functions representing reconstruction losses of the images may be used to learning generative models.
According to one aspect of the present disclosure, a generative adversarial networks-based or GAN-based system is provided for discovering cross-domain relations from two sets of unpaired sample images. The GAN-based system includes a first GAN configured to learn a first generative model for translating images in domain A to images in domain B, and a second GAN configured to learn a second generative model for translating the images in domain B to the images in domain A. The first GAN and the second GAN form a loop such that the sample images in an original domain are translated into a target domain and then translated sample images are reconstructed into the original domain. The first GAN and the second GAN use reconstruction losses of the sample images for learning the first generative model and the second generative model, the reconstruction losses indicating how well the translated sample image has been reconstructed into the original domain.
According to another aspect, a GAN-based system for discovering cross-domain relations from two sets of unpaired sample images includes a first generator configured to learn a first generative model for performing an image-to-image translation from a first domain to a second domain, a first discriminator configured to learn a first discriminative model for discriminating between images translated by the first generator and sample images representing the second domain, a second generator configured to learn a second generative model for performing an image-to-image translation from the second domain to the first domain, and a second discriminator configured to learn a second discriminative model for discriminating between images translated by the second generator and sample images representing the first domain. The first generator and the second generator form a loop such that the sample images in an original domain are translated into a target domain and then translated sample images are reconstructed into the original domain.
The first generator and the second generator respectively iteratively update the first generative model and the second generative model, respectively, so as to minimize reconstruction losses of the sample images and standard GAN generator losses of the sample images. Here, the reconstruction losses indicate how well the translated sample image has been reconstructed into the original domain, and the standard GAN generator losses indicate how realistic the translated images are in a target domain.
In yet another aspect, the present disclosure provides a method of discovering cross-domain relations from two sets of unpaired sample images. The method includes providing a first GAN to perform an unsupervised learning of a first generative model for translating images in domain A to images in domain B, and a second GAN to perform a unsupervised learning of a second generative model for translating the images in domain B to the images in domain A. Here, the first GAN and the second GAN form a loop such that the sample images in an original domain are translated into a target domain and then translated sample images are reconstructed into the original domain. The method uses reconstruction losses in to train the first generative model and the second generative model, the reconstruction losses indicating how well the translated sample image has been reconstructed into the original domain.
In yet another aspect, the present disclosure provides a method of discovering cross-domain relations from two sets of unpaired sample images. The method utilizes a first GAN to perform an unsupervised training of a first generative model for translating images in domain A to images in domain B, and a second GAN to perform a second unsupervised training of a second generative model for translating the images in domain B to the images in domain A. The first GAN and the second GAN is coupled to form a loop such that the sample images in an original domain are translated into a target domain and then translated sample images are reconstructed into the original domain. The method also includes using reconstruction losses in to train the first generative model and the second generative model, the reconstruction losses indicating how well the translated sample image has been reconstructed into the original domain.
Some embodiments of the present disclosure are described below with reference to the accompanying exemplary drawings. In the following description, like reference numerals would rather designate like elements, although the elements are shown in different drawings. Further, in the following description of some embodiments, a detailed description of known functions and configurations incorporated herein will be omitted for the purpose of clarity and for brevity and not to obscure the subject matter of the present disclosure.
The present disclosure in some embodiments provides a new GAN-based model that can learn cross-domain relations from unlabeled images. The disclosed model in some embodiments is trained with two sets of images without requiring images with explicit labels (i.e., paired examples) and obviates the need for a prior training. The disclosed model is capable of translating images from one domain to another domain by using learned cross-domain relations.
In short, the present disclosure couples two different GANs together. The respective GANs have generators for mapping the respective domains to their counterpart domains. The key intuition of the present disclosure is to constrain all images in one domain to be representable by images in the other domain. For example, when learning to generate a shoe image based on each handbag image, the disclosed model forces this generated image to be an image-based representation of the handbag image (and hence reconstructs the handbag image) through a reconstruction loss, and to be as close to the images in the shoe domain as possible through a GAN loss. These two properties used encourage bilateral one-to-one (rather than many-to-one or one-to-many) mapping between two domains.
The following describes a formularized definition of cross-domain relations and presents the problem of learning to discover such relations in two different domains. Standard GAN model (proposed by Non-patent document 1) and a similar variant model will be cited as to their applicability for this task, which has limitations to be addressed by the present GANs-based architecture for discovering cross-domain relations.
The cross-domain relation is mathematically defined as a function GAB that maps elements from its domain A to elements in its codomain B, and as a function GBA that is similarly defined. In fully unsupervised setting, functions GAB and GBA may be arbitrarily defined. To find a meaningful relation, a condition needs to be imposed on the relation of interest. Some embodiments constrain relation to be a one-to-one correspondence (i.e., bijective mapping), which means that GAB is the inverse mapping of GBA. In other words, the range of function GAB, which is the complete set of all possible resulting values GAB(xA) for all xA's in domain A, needs to be contained in domain B. Likewise, the range of function GBA, which is the complete set of all possible resulting values GBA(xB) for all xB's in domain B, needs to be contained in domain A.
The present disclosure relates these constraints to objective functions. Ideally, the equality GBA∘GAB(xA)=xA is satisfied, but this hard constraint is difficult to optimize and a relaxed soft constraint is more desirable in the view of optimization. For this reason, present disclosure provides a model which minimizes the distance d(GBA∘GAB(xA), xA), where any form of metric function (L1, L2, Huber loss, etc.) can be used. Similarly, the disclosed model is designed to minimize d(GAB∘GBA(xB), xB).
Guaranteeing that GA maps to domain B is also very difficult to optimize. The present disclosure relaxes this constraint by minimizing two generative adversarial losses represented by Equation 1 and Equation 2.
−x
−x
Here, is an expectation function, and xA and xB are samples of domain A and B, respectively.
The following describes some GAN architectures to be trained with these loss functions. The description for GAN architectures below will use the following notation.
For example, a generator as denoted by GA is represented by Equation 3 below. Subscripts denote the input and output domains and superscripts denote the input and output image sizes. A discriminator as denoted by DB is represented by Equation 4 below. In Equation 4 that is [0, 1], subscript B denotes that it discriminates images in domain B. Equations 3 and 4 assume that domains A and B have images of size 64×64×3.
GAB:A64×64×3→B64×64×3 Equation 3
DB:B64×64×3→[0,1] Equation 4
Notations GBA and DA are used for the other generator and the other discriminator similar to Equations 3 and 4.
Each generator can be seen as an encoder-decoder pair. The generator may have an encoder section composed of convolution layers with 4×4 filters, each followed by a leaky ReLU (Rectified Linear Unit). The generator may also have a decoder section composed of deconvolution layers with 4×4 filters, followed by a ReLU, and it finally outputs a target domain image of size 64×64×3. The numbers of the convolution and deconvolution layers may range from four to five, depending on the domain. In addition, the dimensions of the convolution and deconvolution layers may be appropriately modified. For detailed descriptions of ReLU and leaky ReLU, Non-Patent Documents 2 and 3 may be consulted.
The discriminator is similar to the encoder section of the generator. Besides the convolution layers and leaky ReLUs, the discriminator has an additional convolution layer with 4×4 filters, and a final sigmoid for outputting a scalar output between [0, 1].
A standard GAN is supposed to take a random Gaussian noise z, encode it into hidden features h and generate images such as MNIST digits. A slight modification needs to be made to this model to fit the relation discovery task: the modified model takes in an image as input instead of noise, as shown in
Different from the architecture of
Generator GAB translates input image xA from domain A into xAB in domain B as represented by Equation 5. The generated image xAB is then translated into a domain-A image xABA to match the original input image xA as represented by Equation 6. Various forms of distance functions, such as MSE, cosine distance, and hinge-loss, can be used as the reconstruction loss d as represented by Equation 7. The translated output xAB is then scored by discriminator DB which compares the translated output xAB to a real sample xB in domain B.
xAB=GAB(xA) Equation 5
xABA=GBA(xAB)=GBA∘GAB(xA) Equation 6
LCONST
LGAN
Generator GAB receives two types of losses as in Equation 9. A first type is a standard GAN generator loss LGAN
TG
LD
−x
During training, the generator GAB learns the mapping from domain A to domain B under two relaxed constraints: that domain A maps to domain B, and that the mapping on domain B is reconstructed into domain A. However, this model lacks a constraint on mapping from B to A, and these two conditions alone does not guarantee a cross-domain relation because the mapping satisfying these constraints is one-directional. In other words, the mapping is an injection, not bijection, and one-to-one correspondence is not guaranteed.
Considered are the two possibly multi-modal image domains A and B.
In some sense, the architecture of
With reference to
The architecture of
The generated images xAB and xBA from generators GAB and GBA are each fed into separate discriminators DB and DA, respectively.
The model of
As a result of coupling two GAN models, the total generator loss is the sum of GAN losses and reconstruction losses for the respective partial models as presented by Equation 11. Similarly, the total discriminator loss LD is a sum of discriminator losses of the two discriminators DA and DB which discriminate real from fake images in domain A and domain B as presented by Equation 12.
L
D
=L
D
+L
D
Equation 12
Now, the present model is constrained by two LGAN losses and two LCONST losses. This achieves a bijective mapping, enabling the discovery of a one-to-one correspondence defined by the present disclosure as cross-domain relations.
The generators and the discriminators are alternately and repeatedly trained. For example, in one step, discriminators DB and DA are updated (e.g., changing the weight of the value passed between the deconvolution layers) toward minimizing the discriminator losses, and in the subsequent step, generators GAB and GBA are updated (e.g., changing the weight of the value passed between the deconvolution layers) toward minimizing the total generator loss determined by the updated discriminators and expressed by Equation 11.
First, referring to
Referring to
Calculations for the provision of the sample images, the reconstruction losses and other terms may be carried out by a training module (not shown) having one or more processors and storages. The generators and discriminators are alternately and repeatedly trained, when the set of sample images used in the earlier training stage can be reused in the next training stage, although it is efficient for training to use a new set of sample images.
Neural networks according to the architecture provided by some embodiments of the present disclosure can be trained to discover relations between two different domains from two sets of unlabeled images. Using the discovered relations, the neural networks can transmit styles from one domain to the other domain successfully, while maintaining the key attributes such as orientations, facial identities, etc.
The provided architecture resolves all the issues of mode collapse and mode oscillation.
Architecture provided by the present disclosure is capable of simultaneous training of two reliable image-to-image translators respectively performing cross-domain translations of images by using inputs of two sets of unlabeled images representing image domains, respectively, and it can perform various image translation processing.
The aforementioned GAN-based method of discovering cross-domain relations may be implemented as computer-readable codes in a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices storing data that can be read by a computer system. In other words, the computer-readable recording medium includes a magnetic recording medium (e.g., ROM, floppy disks, hard disks, etc.), and a storage medium such as an optically readable media (e.g., CD-ROM, DVD, etc.). Further, the computer-readable recording medium can be distributed in computer systems connected via a network, and computer-readable codes can be stored and executed in a distributed mode.
Although exemplary embodiments of the present disclosure have been described for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the various characteristics of the disclosure. Therefore, exemplary embodiments of the present disclosure have been described for the sake of brevity and clarity. Accordingly, one of ordinary skill would understand the scope of the disclosure is not limited by the explicitly described above embodiments but by the claims and equivalents thereof. The scope of protection of the present embodiments is to be construed by the following claims, and all the technical ideas within the scope of their equivalents should be construed as being included in the scope of the embodiments.
Number | Date | Country | Kind |
---|---|---|---|
10-2017-0054474 | Apr 2017 | KR | national |
Number | Name | Date | Kind |
---|---|---|---|
20180024968 | Clinchant | Jan 2018 | A1 |
20180247201 | Liu | Aug 2018 | A1 |
20180307947 | Choi | Oct 2018 | A1 |
Entry |
---|
P. Russo, F. M. Carlucci, T. Tommasi, and B. Caputo, “From source to target and back: symmetric bi-directional adaptive GAN”. arXiv Preprint, arXiv:1705.08824, 2017. (Year: 2017). |
M.-Y.Liu, T. Breuel, and J. Kautz. “Unsupervised image-to-image translation networks”. In Neural Information Processing Systems (NIPS), 2017. 2, 5 (Year: 2017). |
T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. “Improved techniques for training gans”. In Neural Information Processing Systems (NIPS), 2016. 2 (Year: 2016). |
J.-Y.Zhu, T. Park, P. Isola, and A. A. Efros. “Unpaired image-to-image translation using cycle-consistent adversarial networks”. In IEEE International Conference on Computer Vision (ICCV), 2017. 1, 2, 7, 8 (Year: 2017). |
T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, and A. A. Efros. “Learning dense correspondence via 3d-guided cycle consistency”. In CVPR, pp. 117-126,2016 (Year: 2016). |
Ian J. Goodfeloow et al., “Generative adversarial nets”, total 9 pages, Advances in Neural Information Processing Systems, 2014. |
Andrew L. Maas et al., “Rectifier nonlinearities improve neural network acoustic models”, total 6 pages, the 30th International Conference on Machine Learning (ICML), 2013. |
Bing Xu et al., “Empirical evaluation of rectified activations in convolution network”, total 5 pages, arXiv preprint arXiv:1505:00853, 2015. |
Taeksoo Kim et al., “Learning to Discover Cross-Domain Relations with Generative Adversarial Networks”, total 10 pages, arXiv.org, Mar. 15, 2017. |
Number | Date | Country | |
---|---|---|---|
20180314716 A1 | Nov 2018 | US |