Patent Application
20220366228
The present disclosure relates to a technical field of a data generation apparatus, a data generation method, a learning apparatus and a recording medium.
A data generation apparatus using a Generative Adversarial Network (GAN) that is configured to generate a fake data (for example, a fake image) that imitates a real data (for example, a real image) is known as a data generation apparatus. The Generative Adversarial Network includes a Generator that generates the fake data and a Discriminator that discriminates the fake data from the real data. A learning of the Generator is performed so that the Generator is configured to generate the fake data that can deceive the Discriminator and a learning of the Discriminator is performed so that the Discriminator is configured to discriminate the fake data generated by the Generator from the real data.
The Generative Adversarial Network is applied to various technical fields. For example, a Patent Literature 1 discloses an ophthalmic image processing apparatus that obtains a high-resolution image from a low-resolution image by using the Generator (specifically, a generation model used by the Generator) that is learned by using the Generative Adversarial Network.
Note that there are Patent Literatures 2 to 3 and Non-Patent Literatures 1 to 3 as a background art document relating to the present disclosure.
The Generative Adversarial Network has such a technical problem that the learning of the Generator and the Discriminator needs enormously much time. Namely, the Generative Adversarial Network has such a technical problem that it is difficult to perform the learning of the Generator and the Discriminator efficiently.
It is an example object of the present disclosure to provide a data generation apparatus, a data generation method and a recording medium that can solve the above described technical problem. By way of example, an example object of the present disclosure is to provide a data generation apparatus, a learning apparatus, a data generation method and a recording medium that is configured to efficiently perform a learning of an apparatus that is configured to perform a learning of a generating unit for generating a fake data and a discriminating unit for discriminating the fake data from a real data.
One example aspect of a data generation apparatus of the present disclosure includes: an obtaining unit that obtains real data; a fake data generating unit that obtains or generates fake data that imitates the real data; and a mix data generating unit that generates mix data by mixing the real data and the fake data at a desired mix ratio, the mix data generating unit changes the mix ratio that is used to generate a data element of the mix data based on a position of the data element in the mix data.
One example aspect of a learning apparatus of the present disclosure includes: an obtaining unit that obtains real data; a fake data generating unit that obtains or generates fake data that imitates the real data; a mix data generating unit that generates mix data by mixing the real data and the fake data at a desired mix ratio; and a discriminating unit that discriminates discrimination target data including the real data, the fake data and the mix data by using a discrimination model, the discriminating unit performs a learning of the discrimination model based on a discriminated result of the discrimination target data by the discriminating unit, the mix data generation unit changes the mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period that includes a period before a predetermined time elapses from a start of a learning of the generation model and the discrimination model is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period and that includes a period after the predetermined time elapses from the start of the learning of the generation model and the discrimination model.
One example aspect of a data generation method of the present disclosure includes: an obtaining step that obtains real data; a fake data generating step that obtains or generates fake data that imitates the real data; and a mix data generating step that generates mix data by mixing the real data and the fake data at a desired mix ratio, the mix ratio that is used to generate a data element of the mix data changing based on a position of the data element in the mix data in the mix data generation step.
One example aspect of a recording medium of the present disclosure is a recording medium on which a computer program that allows a computer to execute a data generation method is recorded, wherein the data generation method includes: an obtaining step that obtains real data; a fake data generating step that obtains or generates fake data that imitates the real data; and a mix data generating step that generates mix data by mixing the real data and the fake data at a desired mix ratio, the mix ratio that is used to generate a data element of the mix data changing based on a position of the data element in the mix data in the mix data generation step.
Next, an example embodiment of a data generation apparatus, a data generation method and a recording medium will be described with reference to the drawings.
Firstly, with reference to
As illustrated in
The arithmetic apparatus 2 includes at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit) and a FPGA (Field Programmable Gate Array), for example. The arithmetic apparatus 2 reads a computer program. For example, the arithmetic apparatus 2 may read a computer program that is stored in the storage apparatus 3. For example, the arithmetic apparatus 2 may read a computer program that is stored in a non-transitory computer-readable recording medium by using a non-illustrated recording medium reading apparatus. The arithmetic apparatus 2 may obtain (namely, download or read) a computer program from a non-illustrated apparatus that is disposed outside the data generation apparatus 1 through a non-illustrated communication apparatus. The arithmetic apparatus 2 executes the read computer program. As a result, a logical functional block for performing an operation that should be performed by the data generation apparatus 1 is implemented in the arithmetic apparatus 2. Namely, the arithmetic apparatus 2 is configured to serve as a controller for implementing the logical block for performing the operation that should be performed by the data generation apparatus 1.
In the present example embodiment, a logical functional block for allowing the data generation apparatus 1 to serve as a data generation apparatus using a Generative Adversarial Network (GAN) is implemented in the arithmetic apparatus 2.
The real data obtaining unit 21 obtains a real image D_real that is usable as leaning data (in other words, training data) for performing a learning of each of the fake data generation unit 22 and the mix data generation unit 23. The real image D_real means an image that should be discriminated by the discrimination unit 23 that it is real (namely, it is not a below described fake image D_fake generated by the fake data generation unit 22). Incidentally, the image shall mean at least one of a still picture and a video in the present example embodiment, when there is no notation. The real image D_real obtained by the real data obtaining unit 21 is inputted to the discrimination unit 23 as a discrimination target image that should be discriminated by the discrimination unit 23.
The fake data generation unit 22 generates the fake image D_fake that imitates the real image D_real. Note that the “fake image D_fake that imitates the real image D_real” means an image that is generated for the purpose of the discrimination unit 23 erroneously discriminating it to be real (namely, the real image D_real). The fake data generation unit 22 generates the fake image D_fake by using a generation model G that is an arithmetic model (in other words, a learnable learning model) that is configured to generate the fake image D_fake, for example. The fake image D_fake generated by the fake data generation unit 22 is inputted to the discrimination unit 23 as the discrimination target image. Note that the fake data generation unit 22 may obtain the fake image D_fake that is already generated, in addition to or instead of generating the fake image D_fake. For example, the e fake image D_fake that is already generated may be stored in the storage apparatus 3 and the fake data generation unit 22 may obtain (namely, read) the fake image D_fake from the storage apparatus 3.
The discrimination unit 23 discriminates the discrimination target image inputted to the discrimination unit 23. Specifically, the discrimination unit 23 discriminates whether the discrimination target image is the real image D_real or not (in other words, the fake image D_fake or not). The discrimination unit 23 discriminates the discrimination target image by using a discrimination model D that is an arithmetic model (in other words, a learnable learning model) that is configured to discriminate the discrimination target image.
A discriminated result of the discrimination target image by the discrimination unit 23 is used for the learning of each of the fake data generation unit 22 and the mix data generation unit 23 (more specifically, a learning of each of the generation model G and the discrimination model D). Specifically, the learning of the generation model G is performed based on the discriminated result of the discrimination target image by the discrimination unit 23 so that the fake data generation unit 22 is configured to generate the fake image D_fake by which the discrimination unit 23 is deceivable (namely, the fake image D_fake that allows the discrimination unit 23 to erroneously discriminate that it is the real image D_fake). On the other hand, the learning of the discrimination model D is performed so that the discrimination unit 23 is configured to discriminate the fake image D_fake from the real image D_real.
As a result of the learning of the generation model G and the discrimination model D, the data generation apparatus 1 can build the generation model G that is configured to generate the fake image D_fake that cannot be easily distinguished from the real image D_real. As a result, the data generation apparatus 1 having the generation model G that is already learned (alternatively, any apparatus using the generation model G that is already learned) is configured to generate the fake image D_fake that cannot be easily distinguished from the real image D_real. The generation model G may be used to generate the image a resolution of which is higher than that of an image inputted to the generation model G, for example. The generation model G may be used to convert (in other words, translate) image inputted to the generation model G into another image, for example.
Especially in the present example embodiment, a mix data generation unit 24 is implemented in the arithmetic apparatus 2 as the logical functional block for allowing the data generation apparatus 1 to serve as the data generation apparatus using the Generative Adversarial Network. The mix data generation unit 24 generates mix data D_mix by mixing the real data D_real and the fake data D_fake. The mix data D_mix is equivalent to an image (namely, the fake image D_fake) that imitates the real data D_real, because the mix data D_mix is different from the real image D_real. Thus, the mix image generation unit 24 may be regarded to generate the fake image D_fake by a method different from that of the fake data generation unit 22. The mix image D_mix generated by the mix data generation unit 24 is inputted to the discrimination unit 23 as the discrimination target image. Therefore, in the present example embodiment, the discrimination unit 24 discriminates whether the mix image D_mix inputted as the discrimination target image is the real image D_real or not (in other words, the fake image D_fake or not).
The storage apparatus 3 is configured to store a desired data. For example, the storage apparatus 3 may temporarily store the computer program that is executed by the arithmetic apparatus 2. The storage apparatus 3 may temporarily store a data that is temporarily used by the arithmetic apparatus 2 when the arithmetic apparatus 2 executes the computer program. The storage apparatus 3 may store a data that is stored for a long term by the data generation apparatus 1. Note that the storage apparatus 3 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk apparatus, a magneto-optical disc, a SSD (Solid State Drive) and a disk array apparatus. Namely, the storage apparatus 3 may include a non-transitory recording medium.
The input apparatus 4 is an apparatus that receives an input of an information from an outside of the data generation apparatus 1 to the data generation apparatus 1.
The output apparatus 5 is an apparatus that outputs an information to an outside of the data generation apparatus 1. For example, the output apparatus 5 may output an information relating to the learning operation performed by the data generation apparatus 1. For example, the output apparatus 5 may output an information relating to the generation model G that is learned by the learning operation.
Next, with reference to
As illustrated in
After, before or in parallel with the operation at the step S11, the fake data generation unit 22 generates the fake image D_fake (a step S12). The fake data generation unit 22 generates the fake image D_fake by using the generation model G as described above. The generation model G is an arithmetic model that outputs the fake image D_fake based in an inputted random number when the random number (in other words, a noise or a seed) is inputted thereto. The generation model G is an arithmetic model that includes a Neural Network, however, may be other type of arithmetic model. Note that the fake data generation unit 22 typically generates a plurality of fake images D_fake, however, may generate single fake image D_fake.
Then, the mix data generation unit 24 generates the mix image D_mix by mixing the real image D_real obtained at the step S11 and the fake image D_fake obtained at the step S12 (a step S13). For example, as illustrated in
Especially in the present example embodiment, the mix data generation unit 24 may change the mix ratio α(x,y) for generating the pixel D_mix(x,y) based on the coordinate (x,y). Namely, the mix data generation unit 24 may change the mix ratio α by which the real image D_real and the fake image D_fake are multiplied based on the coordinate (x,y). In this case, the mix data generation unit 24 may change the mix ratio α by using a function F in which at least one of the coordinate value x and the coordinate value y is an argument. In other words, the mix data generation unit 24 may set the mix ratio α by using the function F in which at least one of the coordinate value x and the coordinate value y is the argument. Namely, the mix data generation unit 24 may set the mix ratio α by using an equation of α(x,y)=F(x,y). Note that the mix ratio α will be described later in detail with reference to
Then, the discrimination unit 23 discriminates the description target images that include the real image D_real obtained at the step S11, the fake image D_fake generated at the step S12 and the mix image D_mix generated at the step S13 (a step S14). Specifically, the discrimination unit 24 discriminates (in other words, determines) whether the discrimination target image is the real image D_real or not (in other words, is the fake image D_fake or not).
Then, the arithmetic apparatus 2 performs the learning of each of the generation model G and the discrimination model D based on the discriminated result of the discrimination target image by the discrimination unit 23 at the step S14 (a step S15). The arithmetic apparatus 2 may perform the learning of the generation model G and the discrimination model D by using an existing loss function that is used by a learning of the existing Generative Adversarial Network. For example, the arithmetic apparatus 2 may perform the learning of the generation model G and the discrimination model D by using a loss function for achieving such a goal that the fake image D_fake by which the discrimination unit 23 is deceivable can be generated from the generation model G and the fake image D_fake and the real image D_real can be discriminated by the discrimination model D. In this case, the arithmetic apparatus 2 may perform the learning of the generation model G and the discrimination model D by using a loss function including a gradient penalty term disclosed in the above described Non-Patent Literature 3. Moreover, the arithmetic apparatus 2 may perform the learning of each of the generation model G and the discrimination model D by using a learning algorithm such as a backpropagation and the like. Thus, a detailed description of the learning of the generation model G and the discrimination model D is omitted. Note that the arithmetic apparatus 2 may include a learning unit for performing the learning at the step S15 as a processing block.
Then, the arithmetic apparatus 2 determines whether or not the learning operation illustrated in
As a result of the determination at the step S16, when it is determined that the learning operation does not end (the step S16: No), the arithmetic apparatus 2 repeat the operation after the step S11. Namely, the real data obtaining unit 21 obtains new real image D_real that is used for the learning operation (the step S11). The fake data generation unit 22 generates new fake image D_fake by using the generation model G learned at the step S15 (the step S12). The mix data generation unit 24 generates new mix image D_mix by mixing the real image D_real newly obtained at the step S11 and the fake image D_fake newly generated at the step S12 (the step S13). The discrimination unit 23 discriminates new description target images that include the real image D_real newly obtained at the step S11, the fake image D_fake newly generated at the step S12 and the mix image D_mix newly generated at the step S13 (a step S14). The arithmetic apparatus 2 performs the learning of each of the generation model G and the discrimination model D based on the discriminated result of new discrimination target image by the discrimination unit 23 at the step S14 (a step S15).
On the other hand, as a result of the determination at the step S16, when it is determined that the learning operation ends (the step S16: Yes), the arithmetic apparatus 2 ends the learning operation illustrated in
Next, with reference to
As illustrated in
In an example illustrated in
A function using a hyperbolic function is one example of the function F that can change the mix ratio α in this manner. For example,
When the mix image D_mix is generated by using this mix ratio α, the mix image D_mix includes an image part I_fake in which the fake image D_fake is dominant, an image part I_real in which the real image D_real is dominant and an image part I_shift in which the real image D_real and the fake image D_fake are balanced, as illustrated in
When the mix ratio α changes in the monotonical and continuous manner based on the coordinate value x as illustrated in
Note that the mix data generation unit 24 may change the mix ratio α in the continuous manner (in other words, smoothly) based on the coordinate value y, although it is not illustrated in the drawing for convenience of description. The mix data generation unit 24 may change the mix ratio α in the monotonous and continuous manner based on the coordinate value y.
When the mix ratio α changes in the monotonous and continuous manner based on the coordinate value y, the image part I_shift is located between the image part I_real and the image part I_fake in the Y axis direction. For example, the mix data generation unit 24 may set the mix ratio α by using a function F1(y)=0.5×(1+tanh(y−y1)) as the function F.
Note that a function F1′(x)=0.5×(1+tanh((x−x1)/Δx)) may be used as the function F instead of the above described function F1(x). In this case, the mix data generation unit 24 can change a width (specifically, a size in the X axis direction) of the image part I_shift by changing a variable number Δx. Specifically, the width of the image part I_shift becomes wider as the variable number Δx becomes larger. Similarly, a function F1′(y)=0.5×(1+tanh((x−y1)/Δy)) may be used as the function F instead of the above described function F1(y). In this case, the mix data generation unit 24 can change the width (specifically, a size in the Y axis direction) of the image part I_shift by changing a variable number Δy. Moreover, even when the functions F1(x) and F1(y) are not used, the mix data generation unit 24 may set the mix ratio α so that the width of the image part I_shift in at least one of the X axis direction and the Y axis direction is a desired width. Moreover, the mix data generation unit 24 may set the mix ratio α so that a width of at least one of the image part I_real and the image part I_fake is a desired width.
As illustrated in
A function using an exponential function is one example of the function F that can change the mix ratio α in this manner. For example,
Even when the mix image D_mix is generated by using this mix ratio α, the mix image D_mix includes the image part I_fake, the image part I_real and the image part I_shift, as illustrated in
Note that the mix data generation unit 24 may increase the mix ratio α in the monotonous manner based on the coordinate value y when the coordinate value y is within a third range and may decrease the mix ratio α in the monotonous manner based on the coordinate value y when the coordinate value y is within a fourth range that is different from the third range, although it is not illustrated in the drawing for convenience of description. When the mix ratio α changes in the monotonous manner based on the coordinate value y, the image part I_shift is located between the image part I_real and the image part I_fake in the Y axis direction. For example, the mix data generation unit 24 may set the mix ratio α by using a function F2(y)=e{circumflex over ( )}(−(y−y2){circumflex over ( )}2) as the function F.
As illustrated in
For example,
Note that the function F3(x,y) described with reference to
As illustrated in
When the mix ratio α is fixed regard less of the coordinate value x, it can be said that at least two mix ratios α(x,y) that correspond to at least two different coordinate values x are the same ratios. For example, in the example illustrated in
Incidentally, in the example illustrated in
When the mix image D_mix is generated by using this mix ratio α, the mix image D_mix includes an image part S_fake that is same as a part of the fake image D_fake, an image part S_real that is same as a part of the real image D_real and an image part S_mix in which a part of the fake image D_fake and a part of the real image D_real are mixed, as illustrated in
Note that the mix data generation unit 24 may change the mix ratio α in the continuous manner based on the coordinate value y when the coordinate value y is a value in a seventh range and may set the mix ratio α to be a fixed value regardless of the coordinate value y when the coordinate value x is a value in a eighth range that is different from the seventh range, although it is not illustrated in the drawing for convenience of description.
As described above, in the present example embodiment, not only the real image D_real and the fake image D_fake but also the mix image D_mix that is generated by mixing the real image D_real and the fake image D_fake are inputted to the discrimination unit 23. As a result, the learning of the generation model G and the discrimination model D is also performed based on not only the real image D_real and the fake image D_fake but also the mix image D_mix. As a result, the learning of the generation model G and the discrimination model D is performed more efficiently, compared to a case where the learning of the generation model G and the discrimination model D is performed without using the mix image D_mix.
Specifically, immediately after the learning of the generation model G and the discrimination model D starts, there is a possibility that the fake image D_fake generated by the fake data generation unit 22 is far from the real image D_real (in other words, is very different from the real image D_real). On the other hand, since the mix image D_mix is generated based on the real image D_real, the mix image D_mix possibly include an image that is similar to the real image D_real to some extent. Thus, the generation model G and the discrimination model D can learn both of the fake image D_fake that is far from the real image D_real and the fake image D_fake that is similar to the real image D_real to some extent (namely, the mix image D_mix) at an early phase of the learning of the generation model G and the discrimination model D. On the other hand, when the mix image D_mix is not generated, the generation model G and the discrimination model D can learn only the fake image D_fake that is far from the real image D_real. Thus, in the present example embodiment, since the generation model G and the discrimination model D can learn the fake image D_fake that is similar to the real image D_real to some extent (namely, the mix image D_mix) at the early phase of the learning, a time necessary for the learning of the generation model G and the discrimination model D is reduced. Namely, the learning of the generation model G and the discrimination model D is performed more efficiently.
Moreover, the mix image D_mix corresponds to an intermediate image between the randomly generated fake image D_fake and the real image D_real. Thus, when the mix image D_mix is inputted to the discrimination unit 23, an adverse effect of the randomness of the fake data generation unit 22 on the discrimination unit 23 is reduced, compared to a case where the mix image D_mix is not inputted to the discrimination unit 23. Namely, an adverse effect of the randomness of the fake image D_fake generated by the fake data generation unit 22 on the discrimination unit 23 is reduced. For this reason as well, the learning of the discrimination model D is performed more efficiently. Note that one example of the adverse effect of the randomness of the fake data generation unit 22 on the discrimination unit 23 is such an adverse effect that the fake data generation unit 22 generates new fake image D_fake the feature of which is absolutely different from that of the fake image D_fake previously generated by the fake data generation unit 22 and thus the discrimination unit 23 forgets the previously learned content by newly learning new fake image D_fake, for example.
In the above described description, the mix data generation unit 24 changes the mix ratio α for generating the mix image D_mix based on the coordinate (x,y) of the pixel D_mix(x,y) of the mix image D_mix. On the other hand, in a first modified example, the mix data generation unit 24 may change the mix ratio α based on an elapsed time from the start of the learning operation illustrated in
For example, the mix data generation unit 24 may set the mix ratio α so that a ratio of the image part I_fake in which the fake image D_fake is dominant to the mix image D_mix is equal to or larger than a ratio of the image part I_real in which the real image D_real is dominant to the mix image D_mix before a predetermined time elapses from a start of the learning operation. Namely, the mix data generation unit 24 may set the mix ratio α so that the ratio of the image part I_fake to the mix image D_mix is equal to or larger than the ratio of the image part I_real to the mix image D_mix at the early phase of the learning of the discrimination model D and the generation model G. As one example, the mix data generation unit 24 may set the mix ratio α to be a ratio that is larger than 0 and smaller than 0.5. In this case, the mix image D_mix that is discriminated not to be the real image D_real relatively easily by the discrimination unit 23 is generated at the early phase of the learning. Namely, the mix image D_mix that is discriminated not to be the real image D_real relatively easily by the discrimination unit 23 is inputted to the discrimination unit 23 as the discrimination target image at the early phase of the learning. Thus, the learning of the discrimination model D is performed more efficiently at the early phase of the learning, compared to a case where the mix image D_mix that is so similar to the real image D_real that it is difficult for the discrimination unit 23 to discriminate it from the real image D_real is inputted to the discrimination unit 23 as the discrimination target image.
On the other hand, after the predetermined time elapses from the start of the learning operation, it is expected that the discrimination accuracy of the discrimination unit 23 improves to some extent. Thus, after the predetermined time elapses from the start of the learning operation, the mix data generation unit 24 may set the mix ratio α so that the ratio of the image part I_real to the mix image D_mix is larger than that before the predetermined time elapses from the start of the learning operation. In this case, the mix data generation unit 24 may set the mix ratio α so that the ratio of the image part I_real to the mix image D_mix becomes larger at the elapsed time from the start of the learning operation becomes longer. As one example, the mix data generation unit 24 may gradually increase the mix ratio α from an initial value that is larger than 0 and smaller than 0.5. As a result, the mix data generation unit 24 generates the mix image D_mix that is closer to (namely, more similar to) the real image D_real as the learning of the discrimination model D and the generation model G progresses. Namely, the mix image D_mix (what we call a hard sample) that is difficult to be discriminated not to be the real image D_real by the discrimination unit 23 is inputted to the discrimination unit 23. As a result, the learning of the discrimination model D (furthermore, the learning of the generation model G that is performed adversarially against the learning of the discrimination model D) is performed more efficiently at the early phase of the learning, compared to a case where the mix image D_mix that is difficult to be discriminated not to be the real image D_real by the discrimination unit 23 is not inputted to the discrimination unit 23.
In the above described description, the mix data generation unit 24 generates the mix image D_mix by mixing the real image D_real and the fake image D_fake. However, the mix data generation unit 24 may generate the mix image D_mix by mixing two different real images D_real. The mix data generation unit 24 may generate the mix image D_mix by mixing two same real images D_real. The mix data generation unit 24 may generate the mix image D_mix by mixing two different fake images D_fake. The mix data generation unit 24 may generate the mix image D_mix by mixing two same fake images D_fake. The mix data generation unit 24 may generate new mix image D_mix by mixing two same mix images D_mix generated as the fake images D_fake by the mix data generation unit 24. The mix data generation unit 24 may generate new mix image D_mix by mixing two different mix images D_mix generated as the fake images D_fake by the mix data generation unit 24. In any cases, the generated mix image D_mix may be regarded to be equivalent to the data (namely, the fake image D_fake) that imitates the real image D_real, because it is data that is different from the real image D_real.
In the above described description, the mix data generation unit 24 generates the mix image D_mix by mixing the real image D_real and the fake image D_fake generated by the fake data generation unit 22. However, the mix data generation unit 24 may generate new mix image D_mix by mixing the real image D_real and the mix image D_mix generated as the fake image D_fake by the mix data generation unit 24. Even in this case, the fact remains that the generated mix image D_mix is generated by mixing the real image D_real and the fake image D_fake (namely, the mix image D_mix generated as the fake image D_fake).
The mix data generation unit 24 may generate the mix image D_mix by using the real image D_real on which a desired image processing is performed. The mix data generation unit 24 may generate the mix image D_mix by using the fake image D_fake on which the desired image processing is performed. In this case, an image processing unit for performing the image processing on at least one of the real image D_real obtained by the real data obtaining unit 21 and the fake image D_fake generated by the fake data generation unit 22 may be implemented in the arithmetic apparatus 2. Note that at least one of a scaling processing, a rotation processing, a noise reduction processing and a HDR (High Dynamic Range) processing is one example of the desired image processing.
In the above described description, the data generation apparatus 1 performs the learning operation using the image. Namely, in the above described description, the real data obtaining unit 21 obtains the real image D_real as real data, the fake data generation unit 22 generates the fake image D_fake as fake data, the mix data generation unit 24 generates the mix image D_mix as mix data, and the discrimination unit 23 discriminates the discrimination target image including the real image D_real, the fake image D_fake and the mix image D_mix as discrimination target data. However, the data generation apparatus 1 may perform the learning operation using any data that is different from the image. Namely, the real data obtaining unit 21 may obtain any type of real data, the fake data generation unit 22 may generate any type of fake data, the mix data generation unit 24 may generate any type of mix data by mixing the real data and the fake data, and the discrimination unit 23 may discriminate the discrimination target data including the real data, the fake data and the mix data. Even in this case, the mix data generation unit 24 may generate the mix data by using an equation of mix data=mix ratio α×real data+(1−mix ratio α)×fake data. In this case, the mix data generation unit 24 may change the mix ratio α based on a position of each of a plurality of data elements, which are obtained by segmentalizing the mix data, in the mix data. Note that “the position of the data element in the mix data” here may indicate “a position of a data element (for example, the pixel), which is obtained by segmentalizing a target object (for example, the image) represented by the mix data by a desired unit (for example, a unit of the pixel) that is determined based on the target object, in the target object represented by the mix data”.
For example, the data generation apparatus 1 may perform the learning operation using a sound. In this case, the real data obtaining unit 21 may obtain, as the real data, a real sound that should be discriminated by the discrimination unit 23 that it is real (namely, it is not a fake sound generated by the fake data generation unit 22). The fake data generation unit 22 may generate, as the fake data, the fake sound that imitates the real sound. The mix data generation unit 24 may generate, as the mix data, a mix sound by mixing the real sound and the fake sound. For example, the mix data generation unit 24 may generate the mix sound by using an equation of mix sound=mix ratio α×real sound+(1−mix ratio α)×fake sound. In this case, the mix data generation unit 24 may change the mix ratio α based on a time corresponding to each of a plurality of sound elements that are obtained by segmentalizing the mix sound along a time axis (namely, a position of each sound element in the mix sound). In this case, “the position of the data element in the mix data” described above corresponds to a time corresponding to the sound element that is obtained by segmentalizing the sound along the time axis (namely, the sound element that represents the sound at a certain time).
In the above described description, the data generation apparatus 1 (the arithmetic apparatus 2) includes the discrimination unit 23. On the other hand, a data generation apparatus 1a (an arithmetic apparatus 2a) in a fifth modified example may not include the discrimination unit 23, as illustrated in
At least a part of or whole of the above described example embodiments may be described as the following Supplementary Notes. However, the above described example embodiments are not limited to the following Supplementary Notes.
A data generation apparatus comprising:
an obtaining unit that obtains real data;
a fake data generating unit that generates fake data that imitates the real data; and
a mix data generating unit that generates mix data by mixing the real data and the fake data at a desired mix ratio,
the mix data generating unit changing the mix ratio that is used to generate a data element of the mix data based on a position of the data element in the mix data.
The data generation apparatus according to the Supplementary Note 1, wherein
the mix data generating unit changes the mix ratio that is used to generate each of a plurality of data elements of the mix data in a continuous manner by using a function in which the position of the data element in the mix data is an argument.
The data generation apparatus according to the Supplementary Note 1 or 2, wherein
the mix data generation unit
sets the mix ratio that is used to generate a first data element of the mix data to be a first ratio;
sets the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and
changes the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, from the first ratio to the second ratio in a continuous manner based on the position of the third data element in the mix data.
The data generation apparatus according to any one of the Supplementary Notes 1 to 3, wherein
the mix data generating unit changes the mix ratio that is used to generate each of a plurality of data elements that are included in one data part of the mix data in a continuous manner by using a function in which the position of the data element in the mix data is an argument.
The data generation apparatus according to any one of the Supplementary Notes 1 to 4, wherein
the mix data generation unit
fixes the mix ratio that is used to generate a plurality of data elements included in a first data part of the mix data to be a third ratio;
fixes the mix ratio that is used to generate a plurality of data elements included in a second data part of the mix data that is different from the first data part to be a fourth ratio that is different from the third ratio; and
changes the mix ratio that is used to generate each of a plurality of data elements included in a third part of the mix data, which is between the first and second data parts, from the third ratio to the fourth ratio in a continuous manner based on the position of the data element in the mix data.
The data generation apparatus according to any one of the Supplementary Notes 1 to 5, wherein
the mix data generating unit changes, among multiple values, the mix ratio that is used to generate each of a plurality of data elements of the mix data by using a function in which the position of the data element in the mix data is an argument.
The data generation apparatus according to any one of the Supplementary Notes 1 to 6, wherein
the mix data generation unit
sets the mix ratio that is used to generate a first data element of the mix data to be a first ratio;
sets the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and
changes, among multiple values from the first ratio to the second ratio, the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, based on the position of the third data element in the mix data.
The data generation apparatus according to any one of the Supplementary Notes 1 to 7, wherein
the mix data generating unit changes, among multiple values, the mix ratio that is used to generate each of a plurality of data elements that are included in one data part of the mix data by using a function in which the position of the data element in the mix data is an argument.
The data generation apparatus according to any one of the Supplementary Notes 1 to 8, wherein
the mix data generation unit
fixes the mix ratio that is used to generate a plurality of data elements included in a first data part of the mix data to be a third ratio;
fixes the mix ratio that is used to generate a plurality of data elements included in a second data part of the mix data that is different from the first data part to be a fourth ratio that is different from the third ratio; and
changes, among multiple values from the third ratio to the fourth ratio, the mix ratio that is used to generate each of a plurality of data elements included in a third part of the mix data, which is between the first and second data parts, based on the position of the data element in the mix data.
The data generation apparatus according to any one of the Supplementary Notes 1 to 9, wherein
the mix data generation unit changes the mix ratio so that the mix data includes a fourth data part in which the real data is dominant, a fifth data part in which the fake data is dominant and a sixth data part in which the real data and the fake data are balanced.
The data generation apparatus according to the Supplementary Note 10, wherein
the mix data generation unit changes the mix ratio so that the sixth data part is located between the fourth data part and the fifth data part.
The data generation apparatus according to any one of the Supplementary Notes 1 to 11, wherein
the mix data generation unit changes the mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period.
The data generation apparatus according to the Supplementary Note 12, wherein
the mix data generation unit
sets the mix ratio in the first period so that a ratio of a fifth data part in which the fake data is dominant to the mix data is equal to or larger than a ratio of a fourth data part in which the real data is dominant to the mix data; and
sets the mix ratio in the second period so that a ratio of the fourth data part to the mix data in the second period is larger than a ratio of the fourth data part to the mix data in the first period.
The data generation apparatus according to the Supplementary Note 12 or 13 further comprising a discriminating unit that discriminates discrimination target data including the real data, the fake data and the mix data,
the fake data generating unit generating the fake data by using a generation model that is learnable based on a discriminated result of the discrimination target data by the discriminating unit and that is for generating the fake data,
the discriminating unit discriminating the discrimination target data by using a discrimination model that is learnable based on the discriminated result of the discrimination target data by the discriminating unit and that is for discriminating the discrimination target data,
the first period including a period before a predetermined time elapses from a start of a learning of the generation model and the discrimination model,
the second period including a period after the predetermined time elapses from the start of the learning of the generation model and the discrimination model.
The data generation apparatus according to any one of the Supplementary Notes 1 to 14, wherein
each of the real data, the fake data and the mix data is data relating to an image,
the data element of the mix data includes a pixel of the image,
the position of the data element in the mix data is a position of the pixel in the image.
The data generation apparatus according to any one of the Supplementary Notes 1 to 15, wherein
the mix data generating unit changes the mix ratio that is used to generate each of a plurality of data elements of the mix data in a discontinuous manner or a stepwise manner by using a function in which the position of the data element in the mix data is an argument.
The data generation apparatus according to any one of the Supplementary Notes 1 to 16, wherein
the mix data generating unit changes the mix ratio so that the mix ratio changes, on a line that connects a first data element to a second element of the mix data, (i) from a fifth ratio that allows a ratio of the real data to the fake data is 1:0 to a sixth ratio that allows the ratio of the real data to the fake data is 1:1 or (ii) from the sixth ratio to the fifth ratio, or (iii) from a seventh ratio that allows the ratio of the real data to the fake data is 0:1 to the sixth ratio or (iv) from the sixth ratio to the seventh ration.
18. A learning apparatus comprising:
an obtaining unit that obtains real data;
a fake data generating unit that obtains or generates fake data that imitates the real data;
a mix data generating unit that generates mix data by mixing the real data and the fake data at a desired mix ratio; and
a discriminating unit that discriminates discrimination target data including the real data, the fake data and the mix data by using a discrimination model,
the discriminating unit allowing the discrimination model to be learned based on a discriminated result of the discrimination target data by the discriminating unit,
the mix data generation unit changing the mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period that includes a period before a predetermined time elapses from a start of a learning of the generation model and the discrimination model is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period and that includes a period after the predetermined time elapses from the start of the learning of the generation model and the discrimination model.
A data generation method comprising:
an obtaining step that obtains real data;
a fake data generating step that obtains or generates fake data that imitates the real data; and
a mix data generating step that generates mix data by mixing the real data and the fake data at a desired mix ratio,
the mix ratio that is used to generate a data element of the mix data changing based on a position of the data element in the mix data in the mix data generation step.
A recording medium on which a computer program that allows a computer to execute a data generation method is recorded,
the data generation method comprising:
an obtaining step that obtains real data;
a fake data generating step that obtains or generates fake data that imitates the real data; and
a mix data generating step that generates mix data by mixing the real data and the fake data at a desired mix ratio,
the mix ratio that is used to generate a data element of the mix data changing based on a position of the data element in the mix data in the mix data generation step.
A computer program that allows a computer to execute a data generation method is recorded,
the data generation method comprising:
an obtaining step that obtains real data;
a fake data generating step that obtains or generates fake data that imitates the real data; and
a mix data generating step that generates mix data by mixing the real data and the fake data at a desired mix ratio,
the mix ratio that is used to generate a data element of the mix data changing based on a position of the data element in the mix data in the mix data generation step.
The present disclosure is allowed to be changed, if desired, without departing from the essence or spirit of the invention which can be read from the claims and the entire specification, and a data generation apparatus, a learning apparatus, a data generation method and a recording medium, which involve such changes, are also intended to be within the technical scope of the present disclosure.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2020/017974 | 4/27/2020 | WO |
