The present invention relates to a text generation apparatus, a text generation training apparatus, a text generation method, a text generation training method, and a program.
Text generation techniques based on neural networks have advanced. Such a generation technique is a technique of receiving text as an input and generating a predetermined target text (for example, a summary) based on a pre-trained neural network model.
For example, NPL 1 proposes a method in which a word-level importance (attention) obtained by multiplying the importance of a text included in an input document and the importance of a word is reflected in text generation.
However, in NPL 1, information to be considered when generating a text is given in the form of an importance score, a length embedding, or the like and cannot be given as text.
The present invention has been made in view of the above points and it is an object of the present invention to make it possible to add information to be considered when generating a text as text.
Thus, to achieve the object, a text generation apparatus includes a content selection unit configured to acquire a reference text based on an input text and information different from the input text and a generation unit configured to generate a text based on the input text and the reference text, wherein the content selection unit and the generation unit are neural networks based on learned parameters.
Information to be considered when generating a text can be added as text.
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
A program that implements processing of the text generation apparatus 10 is provided through a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed on the auxiliary storage device 102 from the recording medium 101 via the drive device 100. However, it is not always necessary to install the program from the recording medium 101 and the program may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files, data, and the like.
The memory device 103 reads the program from the auxiliary storage device 102 and stores it upon receiving an instruction to activate the program. The CPU 104 executes a function relating to the text generation apparatus 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.
The text generation apparatus 10 may include a graphics processing unit (GPU) instead of the CPU 104 or together with the CPU 104.
A source text (an input text) and information different from the input text (a condition or information to be considered in summarizing the source text (hereinafter referred to as “consideration information”)) are input to the text generation apparatus 10 as text. The first embodiment will be described with respect to an example in which the length (the number of words) K of a text (a summary) that the text generation apparatus 10 generates based on a source text (hereinafter referred to as an “output length K”) is adopted as consideration information.
The content selection unit 11 estimates the importance [0, 1] of each word included in the source text. The content selection unit 11 extracts a predetermined number of words (up to a top K word in terms of importance) based on the output length K and outputs the result of concatenating the extracted words as a reference text. The importance is the probability of the word being included in a summary.
The generation unit 12 generates a target text (a summary) based on the source text and the reference text output from the content selection unit 11.
The content selection unit 11 and the generation unit 12 are based on neural networks that execute a text generation task (summarization in the present embodiment). Specifically, the content selection unit 11 is based on Bidirectional Encoder Representations from Transformers (BERT) and the generation unit 12 is based on a Transformer-based pointer generator model of “Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008” (hereinafter referred to as “Reference 1”). Thus, the content selection unit 11 and the generation unit 12 execute processing based on learned values of the training parameters (learned parameters) of the neural networks.
Hereinafter, a processing procedure executed by the text generation apparatus 10 will be described.
In step S101, the content selection unit 11 estimates (calculates) the importance of each word included in a source text XC.
In the present embodiment, the content selection unit 11 uses Bidirectional Encoder Representations from Transformers (BERT). BERT has achieved the state-of-the-art (SOTA) in many sequence tagging tasks. In the present embodiment, the content selection unit 11 divides the source text into words using a BERT tokenizer, a fine-tuned BERT model, and a feed forward network added specifically for the task. The content selection unit 11 calculates the importance pextn of each word xCn based on the following equation. pextn indicates the importance of an n-th word xCn in the source text XC.
where BERT( ) is the last hidden state of a pre-trained BERT.
and b1 are training parameters of the content selection unit 11. σ is a sigmoid function. dbert is the dimensionality of the last hidden state of the pre-trained BERT.
Subsequently, the content selection unit 11 extracts a set (word sequence) of K words in order from a word with the highest importance pextn(S102). Here, K is the output length as described above. The extracted word sequence is output to the generation unit 12 as a reference text.
In step S101, the importance may be calculated as pextwn of the following equation (2). In this case, a set (word sequence) of K words is extracted as a reference text in order from the word with the highest pextwn.
where NSj is the number of words in a j-th text Sj∈XC. By using this weighting, a text-level importance can be incorporated and a more fluid reference text can be extracted as compared with the case in which only the word-level importance pextn is used.
According to the present embodiment, the length of a summary can be controlled according to the number of words in a reference text, regardless of whether equation (1) or equation (2) is used.
Subsequently, the generation unit 12 generates a summary based on the reference text and the source text XC (S103).
Details of step S103 will be described below.
The source text encoding unit 121 receives the source text XC and outputs
where dmodel is the model size of the Transformer.
An embedding layer of the source text encoding unit 121 projects one-hot vectors of words xCn (of size V) onto a dword-dimensional vector array using a pre-trained weight matrix
such as that of Glove (“Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP” (hereinafter referred to as “Reference 2”)).
The embedding layer then uses a fully connected layer to map each dword-dimensional word embedding to a dmodel-dimensional vector and passes the mapped embedding to a ReLU function. The embedding layer also adds positional encoding to the word embedding (Reference 1).
Each Transformer encoder block of the source text encoding unit 121 has the same architecture as that of Reference 1. Each Transformer encoder block includes a multi-head self-attention network and a fully connected feed forward network. Each network applies residual connections.
The reference text encoding unit 122 receives a reference text Xp which is a sequence of top K words in terms of importance. The words in the reference text Xp are rearranged in the order of appearance in the source text. The output of the reference text encoding unit 122 is as follows.
The embedding layer of the reference text encoding unit 122 is the same as the embedding layer of the source text encoding unit 121 except for the input.
Each Transformer decoder block of the reference text encoding unit 122 is almost the same as that of Reference 1. The reference text encoding unit 122 has an interactive alignment layer that performs multi-head attention at the output of the encoder stack in addition to two sub-layers of each encoder layer. Residual connections are applied in the same way as in the Transformer encoder block of the source text encoding unit 121.
The decoding unit 123 receives Mp and the word sequence of a summary Y generated through an autoregressive process. Here, Mpt is used as a guide vector for generating the summary. The output of the decoding unit 123 is as follows.
where tϵT is each decoding step.
An embedding layer of the decoding unit 123 uses a pre-trained weight matrix Wet to map a t-th word yt in the summary Y to Myt. The embedding layer concatenates Myt and Mpt and delivers the result to a highway network (“Rupesh Kumar Srivastava, Klaus Greff, and Jurgen Schmidhuber. 2015. Highway networks. CoRR, 1505.00387.”). Thus, the concatenated embedding is as follows.
Wmerge is mapped to a model-dimensional vector and passes through a ReLU as in the source text encoding unit 121 and the reference text encoding unit 122. Positional encoding is added to the mapped vector.
Each Transformer decoder block of the decoding unit 123 has the same architecture as that of Reference 1. This component is used stepwise during testing, such that a subsequent mask is used.
Using a pointer-generator, the synthesis unit 124 selects information from any of the source text and the decoding unit 123 based on copy distributions and generates a summary based on the selected information.
In the present embodiment, a first attention head of the decoding unit 123 is used as a copy distribution. Thus, a final vocabulary distribution is as follows.
where the generation probability is defined as follows.
p(yt|zt=1,y1:t-1,x) is a copy distribution. p(zt) is a copy probability representing a weight as to whether yt is copied from the source text. p(zt) is defined as follows.
The estimation of the importance in step S101 of
Next, training will be described.
The text generation apparatus 10 for training further includes a parameter learning unit 13. The parameter learning unit 13 is implemented by a process of causing the CPU 104 or the GPU to execute one or more programs installed in the text generation apparatus 10.
For training of the content selection unit 11, for example, pseudo-training data such as that of “Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In EMNLP, pages 4098-4109” (hereinafter referred to as “Reference 3”) is used as training data. The training data includes pairs (xCn, rn) of words xCn and labels rn of the entire source text xCn. rn is 1 if xCn is selected for a summary. To automatically create such pairs of data, first, an Oracle source text Soracle that maximizes the ROUGE-R score is extracted in the same way as in “Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In ACL (1), pages 132-141.” Then, a dynamic programming algorithm is used to calculate the word-by-word alignment between a reference summary and Soracle. Finally, all aligned words are labeled 1 and the other words are labeled 0.
Training Data for Generation Unit 12 For training of the generation unit 12, it is necessary to create 3-tuple data (XC, Xp, Y) of a source text, a gold set of extracted words, and a target text (summary). Specifically, for training of the generation unit 12, first, the content selection unit 11 is used to select an Oracle text Soracle and pextn is scored for all words xCn of Soracle. Specifically, the content selection unit 11 is used to select an Oracle text Soracle and pextn is scored for all words xCn of Soracel. Next, top K words are selected according to pextn. The original order of words is maintained at Xp. K is calculated using a reference summary length T. To obtain a natural summary that is close to a desired length, the reference summary length T is quantized into discrete size intervals. In the present embodiment, the size interval is set to 5.
Because the process executed by the content selection unit 11 is a simple binary classification task, a binary cross-entropy loss is used.
where M is the number of training examples.
A main loss for the generation unit 12 is a cross-entropy loss.
Further, attention guide losses for the reference text encoding unit 122 and the decoding unit 123 are added. These attention guide losses are designed to guide an estimated attention distribution to a reference attention.
p(asumt) and p(asall) are the top attention heads of the decoding unit 123 and the reference text encoding unit 122, respectively. n(t) indicates the absolute position in the source text corresponding to the t-th word in the summary word sequence.
The overall loss for the generation unit 12 is a linear combination of the above three losses.
λ1 and λ2 were set to 0.5 in an experiment which will be described below.
Then, the parameter learning unit 13 evaluates processing results of the content selection unit 11 and the generation unit 12 that are based on the training data described above by using the above loss function and updates training parameters of the content selection unit 11 and the generation unit 12 until the loss function converges. Values of the training parameters at which the loss function converges are used as learned parameters.
An experiment performed according to the first embodiment will be described.
The CNN-DM dataset (“Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693-1701” (hereinafter referred to as “Reference 4”)) which is a standard corpus for news summaries was used. Summaries are bullets for articles displayed on websites. Following “Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL (1), pages 1073-1083. (2017),” a non-anonymized version of the corpus was used, each source document was truncated into 400 tokens, and each target summary was truncated into 120 tokens. The dataset contains 286,817 training pairs, 13,368 validation pairs, and 11,487 test pairs. The Newsroom dataset was used to evaluate the domain transfer capability of the model (“Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719. Association for Computational Linguistics.”).
While using the generation unit 12 trained on the CNN/DM dataset, the content selection unit 11 was trained on the Newsroom dataset (Reference 3). Newsroom contains a variety of news sources (38 different news sites). For training of the content selection unit 11, 300,000 training pairs were sampled from all training data. The size of test pairs was 106,349.
The same configuration was used for the two datasets. The content selection unit 11 used a pre-trained BERT Large model (“Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional Transformers for language understanding. CoRR.”). The BERT was fine-tuned for two epochs. Default settings were used for other parameters for fine-tuning. The content selection unit 11 and the generation unit 12 used pre-trained 300-dimensional GloVe embeddings. The model size dmodel of the Transformer was set to 512. The Transformer includes four Transformer blocks for the source text encoding unit 121, the reference text encoding unit 122, and the decoding unit 123. The number of heads was 8 and the number of dimensions of the feed forward network was 2048. The dropout rate was set to 0.2. An Adam optimizer with β1=0.9, β2=0.98, and ε=e−9 (“Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR)”) was used for optimization. Following Reference 1, the learning rate was changed during training. The number of warm-up steps was set to 8,000. The size of the input vocabulary was set to 100,000 and the size of the output vocabulary was set to 1,000.
Table 1 shows ROUGE scores of NPL 1 and the first embodiment.
According to Table 1, it can be seen that the first embodiment outperforms NPL 1 in all aspects of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L).
According to the first embodiment, information to be considered when generating a text (the output length) can be added as text as described above. As a result, a source text (an input text) can be treated equivalently to features of the information to be considered.
In addition, the length is controlled using the length embedding in “Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1328-1338. Association for Computational Linguistics.” In this method, the importance of a word according to the length cannot be explicitly taken into consideration and information to be included in an output text cannot be appropriately controlled in the control of the length. On the other hand, according to the present embodiment, it is possible to more directly generate a highly accurate summary while controlling important information according to the output length K without using the length embedding.
Next, a second embodiment will be described. The second embodiment will be described with respect to points different from the first embodiment. Points not particularly mentioned in the second embodiment may be the same as those in the first embodiment.
The second embodiment differs in the configuration of the generation unit 12 as described above. Thus, the second embodiment also differs from the first embodiment in the procedure for generating a summary based on the reference text and the source text XC in step S103.
First, an embedding layer of the joint encoding unit 125 projects one-hot vectors of words XC1 (of size V) onto a dword-dimensional vector array using a pre-trained weight matrix
such as that of Glove (Reference 2).
The embedding layer then uses a fully connected layer to map each dword-dimensional word embedding to a dmodel-dimensional vector and passes the mapped embedding to a ReLU function. The embedding layer also adds positional encoding to the word embedding (Reference 1).
Transformer encoder blocks of the joint encoding unit 125 encode the embedded source and reference texts as a stack of Transformer blocks. Each Transformer encoder block has the same architecture as that of Reference 1. The Transformer encoder block includes two subcomponents, a multi-head self-attention network and a fully connected feed forward network. Each network applies residual connections. In this model, both the source text and the reference text are individually encoded in the encoder stack. The encoding outputs of the source text and the reference text are represented respectively by
Transformer dual encoder blocks in the joint encoding unit 125 calculate the interactive attention between the encoded source and reference texts. Specifically, the Transformer dual encoder blocks first encode the source and reference texts and then perform multi-head attention on the other outputs of the encoder stack (that is, ECs and EPS). The outputs of the dual encoder stack of the source and reference texts are represented respectively by
An embedding layer of the decoding unit 123 receives the word sequence of a summary Y generated through an autoregressive process. At each decoding step t, the decoding unit 123 projects one-hot vectors of words yt in the same way as the embedding layer of the joint encoding unit 125.
Each Transformer decoder block of the decoding unit 123 has the same architecture as that of Reference 1. This component is used stepwise during testing, such that a subsequent mask is used. The decoding unit 123 uses a stack of decoder blocks to perform multi-head attention on a representation Mp obtained by encoding the reference text. The decoding unit 123 uses another stack of decoder blocks to perform multi-head attention on a representation MC obtained by encoding the source text, on top of the first stack. The first stack is to rewrite the reference text and the second is to complement the rewritten reference text with the original source information. The output of the stacks is
Using a pointer-generator, the synthesis unit 124 selects information from any of the source text, the reference text, and the decoding unit 123 based on copy distributions and generates a summary based on the selected information.
The copy distributions of the source text and the reference text are as follows.
where αptk and αCtn are the first attention head of the last block of the first stack of the decoding unit 123 and the first attention head of the last block of the second stack of the decoding unit 123, respectively.
A final vocabulary distribution is as follows.
Next, training will be described. The parameter learning unit 13 functions during training in the same way as in the first embodiment.
Training data for each of the content selection unit 11 and the generation unit 12 may be the same as in the first embodiment.
Because the process executed by the content selection unit 11 is a simple binary classification task, a binary cross-entropy loss is used.
where M is the number of training examples.
A main loss for the generation unit 12 is a cross-entropy loss.
Further, attention guide losses for the decoding unit 123 are added. These attention guide losses are designed to guide an estimated attention distribution to a reference attention.
αprotot,n(t) is the first attention head of the last block of the joint encoder stack for the reference text. n(t) indicates the absolute position in the source text corresponding to the t-th word in the summary word sequence.
The overall loss for the generation unit 12 is a linear combination of the above three losses.
λ1 and λ2 were set to 0.5 in an experiment which will be described below.
Then, the parameter learning unit 13 evaluates processing results of the content selection unit 11 and the generation unit 12 that are based on the training data described above by using the above loss function and updates training parameters of the content selection unit 11 and the generation unit 12 until the loss function converges. Values of the training parameters at which the loss function converges are used as learned parameters.
An experiment performed according to the second embodiment will be described. Datasets used in the experiment of the second embodiment were the same as those of the first embodiment.
Table 2 shows ROUGE scores of NPL 1 and the second embodiment.
According to Table 2, it can be seen that the second embodiment outperforms NPL 1 in all aspects of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L).
According to the second embodiment, the same advantages as those of the first embodiment can be achieved as described above.
Further, according to the second embodiment, words included in a reference text can also be used to generate a summary.
Next, a third embodiment will be described. The third embodiment will be described with respect to points different from the first embodiment. Points not particularly mentioned in the third embodiment may be the same as those in the first embodiment.
The third embodiment will be described with respect to an example in which summarization in consideration of external knowledge is possible and important information in an input text can be directly controlled according to the external knowledge. Here, in the third embodiment, information similar to a source text is retrieved from a knowledge source database (DB) 20 that stores external knowledge which is a text format document (a set of texts) and K texts in the retrieved information which have high relevance to the source text and a relevance measure indicating the degree of relevance as a reference text are used.
In
In step S201, the search unit 14 searches the knowledge source database 20 using a source text as a query.
In any case, it is assumed that a large amount of knowledge (information) is stored in the knowledge source database 20.
In step S201, the search unit 14 searches the knowledge source database 20 for a group of documents, the number of which is a re-rankable number K′ (about 30 to 1000) which will be described later, using a high-speed search module such as Elasticsearch.
When the knowledge source database 20 is configured as illustrated in (1), any of the search methods, search based on the similarity between the source text and headlines, search based on the similarity between the source text and news articles, or search based on the similarity between the source text and news articles+headlines, can be considered.
On the other hand, when the knowledge source database 20 is configured as illustrated in (2), search based on the similarity between the source text and headlines can be considered. The similarity is a known index for evaluating the similarity between documents such as the number of same words included or the cosine similarity.
In the present embodiment, in any of the cases (1) and (2), it is assumed that K′ headlines are obtained as search results based on the similarity and the headlines are each a text. Hereinafter, K′ texts (headlines) which are the search results are each referred to as a “knowledge source text.”
Subsequently, for each knowledge source text, the content selection unit 11 calculates a text-level relevance measure (a relevance measure of each knowledge source text) using a relevance measure calculation model which is a pre-trained neural network (S202). The relevance measure calculation model may form a part of the content selection unit 11. The relevance measure is an index indicating the degree of relevance, similarity, or correlation with the source text and corresponds to the importance in the first or second embodiment.
A matching network takes the vector array of the source text and the vector array of the knowledge source text as inputs and calculates a text-level relevance measure β (0≤β≤1) for the knowledge source text. For example, a co-attention network (“Caiming Xiong, Victor Zhong, Richard Socher, DYNAMIC COATTENTION NETWORKS FOR QUESTION ANSWERING, Published as a conference paper at ICLR 2017”) may be used as the matching network.
The relevance measure calculation model calculates the text-level relevance measure β by a weighted sum of word-level relevance measures pi. Thus, β=Σwipi (where i=1, . . . , number of words). wi is a learnable parameter of the neural network.
The process described with reference to
Subsequently, the content selection unit 11 extracts, as a reference text, the result of concatenating a predetermined number (K) of two or more knowledge source texts in descending order of the relevance measure β calculated using the method as illustrated in
Subsequently, the generation unit 12 generates a summary based on the reference text and the source text (S204). Details of processing executed by the generation unit 12 may be basically the same as those of the first or second embodiment. However, the probability αptk of attention to each word of the reference text may be weighted as follows using the word-level relevance measure or the text-level relevance measure. In the above description, the variable αptk is defined as an attention head. However, because reference is made to the value of αptk here, αptk corresponds to the attention probability. In the following, the text-level relevance measure or the word-level relevance measure will be represented by β for convenience. Either the word-level relevance measure or the text-level relevance measure may be used or both may be used.
When the text-level relevance measure is used, the attention probability αptk is updated, for example, as follows.
The left side is the attention probability after the update. βS(k) is the relevance measure β of a text S including a word k.
When the word-level relevance measure is used, the word-level relevance measure pi corresponding to the word k is applied to βS(k) in the above equation. When both are used, it is conceivable, for example, to weight the word-level relevance measure by the text-level relevance measure as in the above equation (2). pi is calculated for each text S. pi plays the same role as the importance (of equation (1)) in the first embodiment. Further, k is a word number assigned to each word in the reference text (the set of extracted texts S).
Next, training will be described.
In the third embodiment, training of the content selection unit 11 and the generation unit 12 may be basically the same as in each of the above embodiments. Here, two methods for training the relevance measure calculation model used in the third embodiment will be described.
The first is a method of defining correct information of a text-level relevance measure β from a calculation result of the relevance measure β and a correct target text by using a score such as a Rouge score.
The second is a method of determining correct information of a word-level relevance measure as 1 or 0 indicating whether or not a correct text (for example, a target text such as a summary) includes a corresponding word.
Although an example in which the text generation apparatus 10 includes the search unit 14 has been shown above, all knowledge source texts included in external knowledge included in the knowledge source database 20 may be input to the content selection unit 11 when the external knowledge has been narrowed down in advance. In this case, the text generation apparatus 10 does not have to include the search unit 14.
According to the third embodiment, a summary including words that are not in a source text can be efficiently generated using external knowledge as described above. The words are those included in knowledge source texts which are each text as the name indicates. Thus, according to the third embodiment, information to be considered when generating a text can be added as text.
In a technology disclosed in “Ziqiang Cao, Wenjie Li, Sujian Li, and FuruWei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 152-161. Association for Computational Linguistics,” (1) words included in external knowledge cannot be used for text generation as they are, although a target text can be generated taking into consideration external knowledge. In addition, in this technology, (2) the importance of each content of external knowledge cannot be taken into consideration. On the other hand, in the third embodiment, (1) text-level and word-level relevance measures of external knowledge can be taken into consideration and (2) important parts in external knowledge can be included in an output text using CopyNetwork (the synthesis unit 124).
Next, a fourth embodiment will be described. The fourth embodiment will be described with respect to points different from the first embodiment. Points not particularly mentioned in the fourth embodiment may be the same as those in the first embodiment.
The content selection unit 11 includes M layers of Transformer encoder blocks (Encodersat) and 1 linear transformation layers. The content selection unit 11 calculates an importance pextn of an n-th word xCn in a source text XC based on the following equation.
Where Encodersat( ) represents an output vector of a final layer of Encodersal. W1, b1, and σ are as described with respect to Math. 1.
The content selection unit 11 extracts K words from a source text XC as a reference text Xp in descending order of the importance pextn obtained by inputting the source text XC to the Encodersal. Here, the order of words in the reference text Xp maintains the order in the source text XC. The content selection unit 11 inputs the reference text Xp to a generation unit 12. That is, in the fourth embodiment, a word sequence that the content selection unit 11 has extracted based on a predicted value of the importance of each word is explicitly given to the generation unit 12 as additional text information.
The encoding unit 126 includes M layers of Transformer encoder blocks. The encoding unit 126 receives XC+Xp as an input. XC+Xp is a character sequence with a special token representing a delimiter inserted between XC and Xp. XC+Xp is a string of characters into which a special token representing a delimiter is inserted between Xc and Xp. In the present embodiment, RoBERTa (“Y. Liu et al. arXiv, 1907.11692, 2019.”) is used as an initial value for Encodersal. The encoding unit 126 receives the input XC+Xp and obtains a representation
as a processing result of the M layers of blocks. Here, N is the number of words included in XC. However, in the fourth embodiment, N+K is substituted for N, where K is the number of words included in Xp. Here, the encoding unit 126 includes a self-attention and a two-layer feedforward network (FFN). HMe is input to a context-attention of the decoding unit 123 which will be described later.
The decoding unit 123 may be the same as that of the first embodiment. That is, the decoding unit 123 includes M layers of Transformer decoder blocks. The decoding unit 123 receives the output HMe of the encoder and a sequence {y1, . . . , yt-1} up to one step before output by the model as inputs and obtains a representation
as a processing result of the M layers of blocks. In each step t, the decoding unit 123 linearly transforms hMdt into the dimensionality of vocabulary size V and outputs yt which maximizes the probability as a next token. The decoding unit 123 includes a self-attention (a block bi in
All attention processing in the multi-head attention Transformer blocks uses multi-head attention (“A. Vaswani et al. In NIPS, pages 5998-6008, 2017”). This processing involves a combination of k attention heads and is expressed as Multihead(Q, K, V)=Concat(head1, . . . , headk)W°. Each head is headi=Attention(QWQi, KWKi, VWVi). Here, in the self-attention of an m-th layer of each of the encoding unit 126 and the decoding unit 123, the same vector representation Hm is given to Q, K, and V, and in the content-attention of the decoding unit 123, Q is given to Hmd and HMe is given to K and V.
A weight matrix A in the attention of each head
is expressed by the following equation:
where
Next, training will be described.
Training data of the generation unit 12 may be the same as that of the first embodiment. A loss function of the generation unit 12 is defined as follows using cross entropy. M represents the number of training data examples.
Supervised learning becomes possible by giving pseudo-correct answers because the content selection unit 11 outputs a prediction pextn for each word position n of the source text. Generally, correct answers of summaries are only pairs of a source text and a summary, such that a correct answer of 1 or 0 is not given for each token of the source text. However, assuming that words included in both a source text and a summary are important words, pseudo-correct answers of important tokens can be created by aligning token sequences of both the source text and the summary (“S. Gehrmann et al. In EMNLP, pages 4098-4109, 2018”).
A loss function of the content selection unit 11 is defined as follows using binary cross entropy.
where rmn represents a correct answer of the importance of an n-th word of the source text in an m-th data example.
In the fourth embodiment, the parameter learning unit 13 trains the content selection unit 11 using training data of the importance and trains the generation unit 12 using correct answer information of the summary Y. That is, training of the content selection unit 11 and training of the generation unit 12 are performed independently.
The parameter learning unit 13 performs training with the loss function Lgen=Lencdec+Lsal such that Lgen is minimized.
In the first embodiment, “the importance of each word according to the output length” is learned by making the length of the target text (summary) in the training data equal to the output length which is consideration information. On the other hand, in the fourth embodiment, the training data does not include the output length which is consideration information. Thus, the output length is not particularly taken into consideration in the calculation of the importance of the word in the content selection unit 11 and the importance is calculated only from the viewpoint of “whether or not the word is important for summarization.”
However, also in the fourth embodiment, the summary length can be controlled by taking the output length as consideration information as an input and determining the number of tokens to be extracted based on the output length.
Next, a fifth embodiment will be described. The fifth embodiment will be described with respect to points different from the fourth embodiment. Points not particularly mentioned in the fourth embodiment may be the same as those in the fourth embodiment.
A content selection unit 11 in the fifth embodiment calculates a text-level importance pextSj using a word-level importance pextn obtained from the separately trained Encodersal and outputs a combination of top P texts in terms of pextSj in the source text XC to a generation unit 12 as an input text Xs. The text-level importance pextSj can be calculated based on Math. 3.
An encoding unit 126 in the fifth embodiment receives the input text Xs as an input as illustrated in
Other points are the same as those in the fourth embodiment.
Next, a sixth embodiment will be described. The sixth embodiment will be described with respect to points different from the fourth embodiment. Points not particularly mentioned in the fourth embodiment may be the same as those in the fourth embodiment.
The MT model (a) additionally uses correct answer data on the importance pextn and trains the content selection encoding unit 127 and the decoding unit 123 (that is, the generation unit 12) at the same time. That is, an importance model and a text generation model are trained at the same time. This point is common to the SE and SA models and models which will be described below (that is, those other than the CIT and SEG models). In the MT model, the Encodersal of the content selection encoding unit 127 shares parameters of the encoding unit 126 of the fourth embodiment. In the MT model, only the encoding result (HMe) is input to the decoding unit 123 as is apparent from
The decoding unit 123 may be the same as that of the fourth embodiment.
The SE model (b) biases the encoding result of the content selection encoding unit 127 using the importance pextn (“Q. Zhou et al. In ACL, pages 1095-1104, 2017”). Specifically, the decoding unit 123 weights the encoding result hMen, which is a final output of the encoder blocks of the content selection encoding unit 127, by the importance as follows.
In the SE model (b), hMen input to the decoding unit 123 is replaced with h˜Men (where h˜ indicates a symbol h with ˜ above it). While BiGRU is used in “Q. Zhou et al. In ACL, pages 1095-1104, 2017,” the decoding unit 123 is implemented using a Transformer for fair comparison in the present embodiment. In the SE model, the Encodersal of the content selection encoding unit 127 shares parameters of the encoding unit 126 of the fourth embodiment. In the SE model, only the encoding result (HMe) and the importance pextn are input to the decoding unit 123 as is apparent from
Unlike the SE model, the SA model (c) weights the attention on the decoding unit 123 side. Specifically, the decoding unit 123 weights an attention probability of each step t (a t-th row of Ai) of a weight matrix of an i-th head of the context-attention by pextn as shown in Math. 38, where the weight matrix is represented by
and the attention probability of each step t is represented by
The copy probability of the pointer generator is weighted using a similar method proposed by Gehrmann et al. (“S. Gehrmann et al. In EMNLP, pages 4098-4109, 2018”). On the other hand, in the present embodiment, calculation using the weighted attention probability is performed in the context-attention calculation of all layers of the decoding unit 123 because the pre-trained seq-to-seq model does not have a copy mechanism. In the MT model, the Encodersal of the content selection encoding unit 127 shares parameters of the encoding unit 126 of the fourth embodiment. In the SA model, only the encoding result (HMe) and the importance pextn are input to the decoding unit 123 as is apparent from
A model that combines the MT model and the SE model or the MT model is also effective in the sixth embodiment.
The SE+MT model additionally uses correct answer data on the importance pextn in training of the SE model, such that simultaneous training with summarization is performed.
The SA+MT model additionally uses correct answer data on the importance pextn in training of the SA model, such that simultaneous training with summarization is performed.
The parameter learning unit 13 of the sixth embodiment trains the content selection encoding unit 127 and the decoding unit 123 at the same time as described above.
In the case of the SE and SA models, the parameter learning unit 13 does not give correct answer information of the importance Pextn and trains the generation unit 12 (the content selection encoding unit 127 and the decoding unit 123) using only correct answer information of the summary Y. In this case, the parameter learning unit 13 performs training with the loss function Lgen=Lencdec such that Lgen is minimized.
On the other hand, in the case of the MT model, the parameter learning unit 13 performs multi-task training of the content selection encoding unit 127 and the generation unit 12 (that is, the content selection encoding unit 127 and the decoding unit 123) using correct answer information of the importance S and correct answer information of the summary Y. That is, when training the content selection encoding unit 127 as an importance model (the content selection unit 11), the parameter learning unit 13 learns a task of predicting the importance pextn from Xc using the correct answer information (indicating whether or not the word is important) of the importance Pextn as teacher data. On the other hand, when training the generation unit 12 (content selection encoding unit 127+decoding unit 123) as a Seq2Seq (encoder-decoder) model, the parameter learning unit 13 learns a task of predicting a summary Y from an input text XC using a correct answer summary for XC as teacher data. During this multi-task training, parameters of the content selection encoding unit 127 are shared by the two tasks. The correct answer information (pseudo-correct answer) of the importance Pextn is as described in the fourth embodiment. In this case, the parameter learning unit 13 performs training with the loss function Lgen=Lencdec+Lsal such that Lgen is minimized.
Next, a seventh embodiment will be described. The seventh embodiment will be described with respect to points different from the fourth embodiment. Points not particularly mentioned in the fourth embodiment may be the same as those in the fourth embodiment.
Such a configuration can be implemented by combining the CIT model (or the SEG model) described above with the SE or SA model (or the MT model). Hereinafter, a combination of the CIT model and the SE model (CIT+SE model) and a combination of the CIT model and the SA model (CIT+SA model) will be described.
In the CIT+SE model, the importance of the SE model
is learned on the SE model unsupervised learning for XC+Xp of the CIT model, to weight
That is, XC+Xp is input to the content selection encoding unit 127 of the SE model (where XC is selected by the content selection unit 11) and the importance pext for XC+Xp is estimated.
On the CIT+SA model,
is learned by unsupervised learning, to weight
as in the CIT+SE model. That is, XC+Xp of the CIT model is input to the content selection encoding unit 127 of the SA model and the importance pext for XC+Xp is estimated.
The process executed by the parameter learning unit 13 in the seventh embodiment may be the same as that in the sixth embodiment.
Experiments performed for the fourth to seventh embodiments will be described.
CNN/DM (“K. M. Hermann et al. In NIPS, pages 1693-1701, 2015.”) and XSum (“S. Narayan et al. In EMNLP, pages 1797-1807, 2018.”) which are representative summarization datasets were used. CNN/DM is summarization data with a high extraction rate of about three texts and XSum is summarization data with a low extraction rate of about one text. Evaluation was performed using ROUGE values that are standardly used in automatic summarization evaluation. The outline of each data is shown in Table 3. The average length of a summary was calculated in units of sub-words into which the dev set of each data is divided with byte-level BPE (“A. Radford et al. Language models are unsupervised multi-task learners. Technical report, OpenAI, 2019.”) of fairseq1.
Each model was implemented using fairseq. Training was performed with seven NVIDIA V100 32 GB GPUs. The same settings as those of “M. Lewis et al. arXiv, 1910.13461, 2019” were used for CNN/DM training. After confirmation on the XSum training from the authors, the parameter update frequency (UPDATE FREQ) regarding gradient accumulation of the mini-batch among the CNN/DM settings was changed to 2. After the accuracy of the dev set was confirmed, it was determined that the number of words K to be extracted by the content selection unit 11 of the CIT model during training was a value in units of bins of five, into which the length of the correct answer summary is divided, on CNN/DM and was a fixed length of 30 on XSum. During evaluation, the average summary length of the dev set was set to K on CNN/DM and was set to 30 on XSum as during training. On XSum, settings were done such that important word sequences do not include duplicates. Methods of setting K during training include a method of setting K to an arbitrary fixed value, a method of setting an arbitrary threshold value and setting K to a value equal to or higher than the threshold value, and a method of setting K dependent on the length L of the correct answer summary. When training is performed with the setting of K dependent on the length L of the correct answer summary, the summary length can be controlled by changing the length of K during test.
Tables 4 and 5 show experimental results (ROUGE values of each model) regarding “Does combining importance models improve the summarization accuracy?”
As shown in Tables 4 and 5, CIT+SE gave the best results on both datasets. First, because the CIT alone improved the accuracy, it can be seen that combining important words is excellent as a method of giving important information to the text generation model (the generation unit 12). Because it was confirmed that combining the CIT with the SE and SA models further improved the accuracy, it is thought that the combination with soft weighting achieves an improvement although noise is included in important words.
Both the SE and SA have improved the accuracy. The MT, SE+MT, and SA+MT which simultaneously learn the correct answer of importance have improved the accuracy on CNN/DM, but decreased the accuracy on XSum. This is thought to be due to the effects of the quality of the pseudo-correct answer of importance. In the case of CNN/DM, it is easy to align words when creating a pseudo-correct answer because summaries are relatively long and highly extractive. On the other hand, in the case of XSum data, alignment is difficult and pseudo-correct answers serve as noise because summaries are short and often described in expressions other than those in source text. Even in the CIT, the improvement in accuracy on XSum is smaller than that on CNN/DM due to these effects. However, it can be said that the CIT is a method that operates robustly because it shows the highest performance on all data with different properties.
“Our fine-tuning” indicate the results of fine-tuning (baseline) by the inventors of the present application. Underlines in Table 5 indicate the highest accuracies among the improvements in the baseline models.
Table 6 shows experimental results regarding “how accurate is token extraction of the importance model alone?”
Table 6 shows the results of evaluating the ROUGE values of top token sequences and summary text selected in terms of the importance pext of the CIT.
It can be seen that important words are properly identified on CNN/DM. It can be seen that the accuracies are high in R1 and R2 and important elements can be extracted at the word level as compared with Presum (“Y. Liu et al. In EMNLP-IJCNLP, pages 3728-3738, 2019”) which is the SOTA in the conventional extractive summarization methods. While there were no significant differences in the importance between words with the internally learned pext of the SE model or SA model, important words can be explicitly taken into consideration in the present model. On the other hand, the accuracy of the importance model on XSum is low as a whole because XSum is data with a low extraction rate. This is thought to be the reason why the improvement in summarization accuracy is lower than that of CNN/DM. Although the method of giving a pseudo-correct answer verified here was particularly effective on data with a high extraction rate, a further improvement in accuracy even on data with a low extraction rate can be expected by improving the accuracy of the importance model.
Although the above embodiments have been described with respect to a summary generation task as an example, each of the above embodiments may be applied to various other text generation tasks.
In each of the above embodiments, the text generation apparatus 10 for training is an example of a text generation training apparatus.
Although embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments and various modifications and changes can be made within the scope of the spirit of the present invention described in the claims.
Number | Date | Country | Kind |
---|---|---|---|
PCT/JP2019/038948 | Oct 2019 | WO | international |
This application is a continuation of U.S. patent application Ser. No. 17/764,186, filed on Mar. 27, 2022, which application is a U.S. National Stage Application filed under 35 U.S.C. § 371 claiming priority to International Patent Application No. PCT/JP2020/008955, filed on 3 Mar. 2020, which application claims priority to and the benefit of International Patent Application No. PCT/JP2019/038948, filed on 2 Oct. 2019, the disclosures of which are hereby incorporated herein by reference in their entireties.
Number | Date | Country | |
---|---|---|---|
Parent | 17764186 | Mar 2022 | US |
Child | 18595400 | US |