Automatized text analysis

Information

  • Patent Grant
  • 11194968
  • Patent Number
    11,194,968
  • Date Filed
    Thursday, May 31, 2018
    6 years ago
  • Date Issued
    Tuesday, December 7, 2021
    3 years ago
  • CPC
  • Field of Search
    • US
    • 704 001000
    • 704 009000
    • 704 010000
    • CPC
    • G06F17/27
    • G06F17/20
    • G06F17/2705
    • G06F17/271
    • G06F17/2715
    • G06F17/272
    • G06F17/2725
    • G06F17/273
    • G06F17/2735
    • G06F17/274
    • G06F17/2745
    • G06F17/275
    • G06F17/2755
    • G06F17/276
    • G06F17/2765
    • G06F17/277
    • G06F17/2775
    • G06F17/278
    • G06F17/2785
    • G06F17/2795
  • International Classifications
    • G06F40/30
    • G06N3/08
    • G06N3/04
    • Term Extension
      27
Abstract
The present invention concerns a text analysis system, the text analysis system being adapted for utilizing a topic model to provide a document representation. The topic model is based on learning performed on a text corpus utilizing hidden layer representations associated to words of the text corpus, wherein each hidden layer representation pertains to a specific word of the text corpus and is based on a word environment including words occurring before and after the specific word in a text of the text corpus.
Description
FIELD OF TECHNOLOGY

The present invention concerns automatic text analysis, in particular in the context of Recurring Neural Networks (RNNs).


BACKGROUND

An important task in automatized text analysis is related to determine context of words, e.g. to determine topic of a text or document, in particular for classifying or retrieving information.


SUMMARY

An aspect relates to providing approaches improving text analysis, in particular with improved content determination.


There is disclosed a text analysis system, the text analysis system being adapted for utilizing a topic model to provide a document representation. The topic model is based on learning performed on a text corpus utilizing hidden layer representations associated to words of the text corpus. Each hidden layer representation pertains to a specific word of the text corpus and is based on a word environment comprising words occurring before and after the specific word in a text of the text corpus.


There is also disclosed a method for performing text analysis. The method comprises utilizing a topic model to provide a document representation, the topic model being based on learning performed on a text corpus utilizing hidden layer representations associated to words of the text corpus, wherein each hidden layer representation pertains to a specific word of the text corpus and is based on a word environment comprising words occurring before and after the specific word in a text of the text corpus.


The invention is particularly suitable for document representation for context determination and/or information retrieval, as the consideration of the encompassing word environment facilitates more successful context identification.


The topic model may be a neural autoregressive topic model.


Such models are particularly suitable for context determination and/or for considering the word environment. An exemplary topic model may be based on DOCNADE (Document Autoregressive Neural Distribution Estimator) or be similar thereto.


In some cases, the document representation may represent and/or comprise a word distribution and/or a word probability distribution. Alternatively, or additionally, the representation may represent and/or comprise a measure of generalization like perplexity, and/or interpretability like topic coherence, and/or applicability, e.g. pertaining to document retrieval. The document representation may be determined based on hidden layer representations, e.g. hidden-state variables for each word of interest.


The system may be adapted to determine, and/or the method may comprise determining, a topic and/or document representation of an input text based on the document representation. The input text may be part of the text corpus, or separate or independent thereof. Such determining may be based on learning performed with the topic model, e.g. to determine a space mapping word distributions and/or texts to topics. Alternatively, or additionally, it may be considered that method and/or system are utilised for Natural Language Processing (NLP). The document representation of the input text may be dependent on document representation determined based on the text corpus.


A Recurring Neural Network (RNN) may be used for learning hidden layer representations. The RNN may be used to learn the topic model.


There is also considered a computer program comprising instructions causing a computer system to perform and/or control a method as described herein.


Also, a storage medium storing a computer program as described herein is considered.


In general, determining a document representation may comprise determining document representations for all texts of a text corpus. Determining a document representation of an input document may be based on the representations for all texts, and/or on learning performed thereon.


The method may especially be carried out, completely or in part, by a computer system. To these ends, the method may be formulated as a computer program product with program code means. The above described system may comprise the computer system. Advantages or features of the method may apply to the method and vice versa.





BRIEF DESCRIPTION

The above-described properties, features and advantages of present invention as well as the way they are achieved will be made clearer and better understandable in the light of the following discussion, making reference to exemplary embodiments shown in accompanying FIGURES, in which



FIG. 1 shows an exemplary schematic of a text analysis system according to the invention.





DETAILED DESCRIPTION

An exemplary text analysis system 100 is shown schematically in FIG. 1. A text corpus may generally comprise a plurality of documents or texts, each of which may comprise a different number of words. The topic model may learn based on each text of the text corpus. For an example text or document v with I words vi=v1 . . . vI, a hidden layer representation or hidden-state vector hi may be determined for one or more, in particular each, word vi, based on a word environment comprising both words vn (n<i) and vm (m>i), in particular all words fulfilling such condition. The first word, i=1, and last word, i=I, will have one-sided word environments. The word environment may be ordered according to the arrangement of words in the text. Utilizing the proposed word environment allows using the full context of a word vi, improving context determination.


In FIG. 1, example words v1 . . . v4 of an exemplary document v may be input into the text analysis system with the topic model. Bold lines identify connections sharing parameters, e.g. W. Observations, indicated with a double circle, for each word vi are multinominal. {circumflex over ({circumflex over (v)})}1, {circumflex over ({circumflex over (v)})}2, {circumflex over ({circumflex over (v)})}3, {circumflex over ({circumflex over (v)})}4 represent autoregressive conditionals p. Connections between each vi and hidden units for determining hidden layer presentations are shared, and each conditional is decomposed in a tree of binary logic regressions 104.


Representations hi may be determined in a language modeling and feed-forward fashion, e.g. computing informed representations, which may be considered latent document representations 102. A scaling factor D may be considered to account for documents of different lengths. A sequence of hidden layer representations may be determined using the text analysis system, in particular the topic model and/or associated RNN.


Exemplary computations may be according to a variant incorporation both vn (n<i) and vm (m>i) contexts in computing h, e.g. according to:

hi(vn,vm)=g(Dc+Σk<iW:,vkk>iW:,vk),  (Eq. 1)

wherein the autoregressive conditional may be expressed as:










p


(



v
i

=

w
|

v
n



,

v
m


)


=



exp


(


b
w

+


U

w
,
:





h
i



(


v
n

,

v
m


)




)






w





exp


(


b

w



+


U


w


,
:





h
i



(


v
n

,

v
m


)




)




.





Eq


(
2
)








Alternative computations may account for vi in addition to vn and vm, e.g. according to:

hi(v)=g(Dc+Σk<iW:,Vkk>iW:,vk+W:,vi).  Eq(3)


The autoregressive conditional may be expressed as:










p


(


v
i

=

w
|
v


)


=



exp


(


b
w

+


U

w
,
:





h
i



(
v
)




)






w





exp


(


b

w



+


U


w


,
:





h
i



(
v
)




)




.





Eq


(
4
)








Above, i may be in [1, . . . , D], hi(v) refers to hi (vn, vm, vi) and p(vi=w|v) refers to p(vi=w|vn, vm, vi).


The autoregressive conditionals may be computed by a neural network for each word vi to learn a representation hi efficiently, e.g. based on a linear transformation followed by a non-linear transformation. The weight W may be the same across all conditionals and tied or related contextual observable, which are indicated as bold lines in FIG. 1.


A binary word tree 104 may be used to decompose the computation of the D conditionals, allowing a time logarithmic dependence of K. Variants with or without D may be considered. Parameters b, c, W, U may be learned by minimizing the average log-likelihood across all documents v with stochastic gradient.


For each conditional p, an own hidden layer hi (vn, vm) or hi(vn, vm, vi) may be computed. H may be the size of each hidden layer h, D the size of the document v, or part of the document to be considered.


The test analysis system 100 may undergo training on a text corpus comprising a plurality of documents or text v. After training, a document representation for a new document v′ may be determined, e.g. a topic thereof. For example, a hidden layer representation for the words vi′ in the new document v′ may be determined based on the learning performed by the text analysis system, considering for example for each observation i′ in [1 . . . D′] for words of v′. The representation may be the last hidden layer representation hD′ extracted from the text analysis system.


Any module or functionality or action described herein may be implemented in software, e.g. as a program module or separate program interacting with another program. A system may generally comprise processing circuitry and/or one or more interfaces and/or memory. It may be considered that a system is implemented as or on a computer, and/or a dedicated analysis device. Processing circuitry may comprise integrated circuitry, e.g. one or more processors and/or controllers, e.g. microcontrollers, and/or one or more FPGAs (Field-Programmable Gate Array) and/or ASICs (Application-Specific Integrated Circuitry). The processing circuitry may comprise, and/or be connected or connectable to one or more memories and/or storage media, e.g. for reading and/or writing and/or accessing data. Processing circuitry may be located on one device, or be distributed. A memory or storage medium may be magnetic and/or optical and/or electrical and/or volatile and/or non-volatile. Examples of memories or storage media comprise flash memory, read-only memory, random access memory, CD/DVD/BlueRay, cache memory, buffer memory, etc.


Even though present invention has been illustrated and explained in detail above with reference to the preferred embodiments, the invention is not to be construed as limited to the given examples. Variants or alternate combinations of features given in different embodiments may be derived by a subject matter expert without exceeding the scope of present invention.

Claims
  • 1. A text analysis system, comprising: a processor;a memory device coupled to the processor; anda computer readable storage device coupled to the processor, wherein the storage device contains program code executable by the processor via the memory device to implement a method comprising:utilizing a topic model to provide a document representation, the topic model being based on learning performed on a text corpus utilizing hidden layer representations associated to words of the text corpus, wherein each hidden layer representation pertains to a specific word of the text corpus and is based on a word environment comprising words occurring before and after the specific word in a text of the text corpus, for determining a context of the specific word;wherein the hidden layer representation is calculated according to a formula, wherein the formula is represented by: hi(vn,vm)=g(Dc+Σk<iW:,vk+Σk>iW:,vk);wherein the hidden layer representation is a single unified context vector that simultaneously accumulates the context and the uses the context to learn text representation in a single step;wherein the topic model is a neural autoregressive topic model and an autoregressive conditional is expressed as:
  • 2. The text analysis system according to claim 1, wherein the document representation represents a word probability distribution.
  • 3. The text analysis system according to claim 1, the system being adapted to determine a topic of an input text based on the document representation.
  • 4. The text analysis system according to claim 1, wherein the system utilises a Recurring Neural Network, RNN, for learning hidden layer representations.
  • 5. A method for performing text analysis, the method comprising utilizing a topic model to provide a document representation, the topic model being based on learning performed on a text corpus utilizing hidden layer representations associated to words of the text corpus, wherein each hidden layer representation pertains to a specific word of the text corpus and is based on a word environment comprising words occurring before and after the specific word in a text of the text corpus, for determining a context of the specific word; wherein the hidden layer representation is calculated according to a formula, wherein the formula is represented by: hi(vn,vm)=g(Dc+Σk<iW:,vk+Σk>iW:,vk);wherein the hidden layer representation is a single unified context vector that simultaneously accumulates the context and the uses the context to learn text representation in a single step;wherein the topic model is a neural autoregressive topic model and an autoregressive conditional is expressed as:
  • 6. The method according to claim 5, wherein the document representation represents a word probability distribution.
  • 7. The method according to claim 5, the method comprising determining a topic of an input text based on the document representation.
  • 8. The method according to claim 5, the method comprising utilizing a Recurring Neural Network, RNN, for learning hidden layer representations.
  • 9. A non-transitory computer program comprising instructions causing a computer system to perform and/or control a method according to claim 5.
  • 10. A non-transitory storage medium storing a computer program according to claim 9.
  • 11. A text analysis system, comprising: a processor;a memory device coupled to the processor; anda computer readable storage device coupled to the processor, wherein the storage device contains program code executable by the processor via the memory device to implement a method comprising:utilizing a topic model to provide a document representation, the topic model being based on learning performed on a text corpus utilizing hidden layer representations associated to words of the text corpus, wherein each hidden layer representation pertains to a specific word of the text corpus and is based on a word environment comprising words occurring before and after the specific word in a text of the text corpus, for determining a context of the specific word;wherein the hidden layer representation is calculated according to a formula, wherein the formula is represented by: hi(v)=g(Dc+Σk<iW:,Vk+Σk>iW:,vk+W:,vi)wherein the hidden layer representation is a single unified context vector that simultaneously accumulates the context and the uses the context to learn text representation in a single step;wherein the topic model is a neural autoregressive topic model and an autoregressive conditional is expressed as:
US Referenced Citations (26)
Number Name Date Kind
9235800 Grothmann Jan 2016 B2
9552548 Brestoff Jan 2017 B1
9715496 Sapoznik Jul 2017 B1
9754219 Brestoff Sep 2017 B1
9754220 Brestoff Sep 2017 B1
9760850 Brestoff Sep 2017 B1
10109275 Henry Oct 2018 B2
10169315 Heckel Jan 2019 B1
10210244 Branavan Feb 2019 B1
10216715 Broderick Feb 2019 B2
10387888 Sapoznik Aug 2019 B2
20080281595 Sakai Nov 2008 A1
20110106743 Duchon May 2011 A1
20150095017 Mnih Apr 2015 A1
20160110343 Kumar Rangarajan Sridhar Apr 2016 A1
20170371861 Barborak Dec 2017 A1
20180165554 Zhang Jun 2018 A1
20180300315 Leal Oct 2018 A1
20180307749 Al Hasan Oct 2018 A1
20190130305 Sivertson May 2019 A1
20190236139 DeFelice Aug 2019 A1
20190294962 Vezer Sep 2019 A1
20190303727 Foroughi Oct 2019 A1
20190325068 Lai Oct 2019 A1
20190370331 Buttner Dec 2019 A1
20190370637 Dunning Dec 2019 A1
Non-Patent Literature Citations (5)
Entry
Larocelle et al., A neural autoregressive topic model, 2012, Neural Information Processing Systems, pp. 1-9 (Year: 2012).
Peters et al., Semi-supervised sequence tagging with biderictional language models, Apr. 29, 2017, arXiv:1705.00108v1, pp. 1-10 (Year: 2017).
Peters et al., Deep contextualized word representations, Mar. 22, 2018, arXiv:1802.05365v2, pp. 1-15 (Year: 2018).
Amiri et al., Learining text pair similarity with context-sensitive autoencoders, ACL, 2016, pp. 1-11 (pp. 1882-1892). (Year: 2016).
Zheng et al., A deep and autoregressive approach for topic modeling of multimodal data, 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38 p. 1056-1069. (Year: 2016).
Related Publications (1)
Number Date Country
20190370331 A1 Dec 2019 US