METHOD AND SYSTEM FOR ASPECT-LEVEL SENTIMENT CLASSIFICATION BY GRAPH DIFFUSION TRANSFORMER

Abstract

System and method for aspect-level sentiment classification. The system includes a computing device, the computing device has a processer and a storage device storing computer executable code. The computer executable code is configured to: receive a sentence having a labeled aspect term and context; convert the sentence into a dependency tree graph; calculate an attention matrix of the dependency tree graph based on one-hop attention between any two nodes of the graph; calculate multi-head attention diffusion for any two nodes from the attention matrix; obtain updated embedding of the graph using the multi-head diffusion attention; classify the aspect term based on the updated embedding of the graph to obtain predicted classification of the aspect term; calculate loss function based on the predicted classification and the ground truth label of the aspect term; and adjust parameters of models in the computer executable code based on the loss function.

Description

FIELD OF THE DISCLOSURE

The present disclosure relates generally to sentiment analysis of text, and more specifically related to methods and systems for aspect-level sentiment classification using a graph diffusion transformer (GDT).

BACKGROUND OF THE DISCLOSURE

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

Aspect-level sentiment classification is a fine-grained sentiment analysis task, which aims to identify the sentiment polarity (e.g., positive, negative, or neutral) of a specific aspect term appearing in a review. A review may be, for example: “Despite a slightly limited menu, everything prepared is done to perfection, ultra fresh and a work of food art.” The sentiment polarity of the aspect terms “menu” and “food” in the review are negative and positive, respectively. This task has many applications, such as assisting customers to make purchase decisions on e-commerce websites.

Recent studies have shown that interactions between the aspect term and its context are crucial to identify the sentiment polarity towards the given aspect term. Most approaches only consider semantic information from the context words and utilize attention mechanism to learn such interactions. However, it has been shown that syntactic information obtained from dependency parsing is very effective in capturing long-range syntactic relations that are obscure from the surface form. Recent successful approaches to learn syntax-aware representations employed graph neural network (GNN) model over dependency trees, which allows the message passing between the aspect term and the context words in a syntactical manner. One of the popular GNN model is graph attention network (GAT). In one layer GAT, GAT computes attention scores only on direct one-hop neighbors. When the aspect term is far away from its opinion context, more GAT layers are needed. For example, in FIG. 1, there's a four-hop distance on the dependency tree between the aspect term “soup” and the opinion words “not tasty.” Therefore, a four-layer GAT is needed in this case. Although deeper GAT could still pass information among distant nodes by stacking many layers, most of previous work already indicated that GNN models with two layers achieve the best performance and deeper GAT do not bring additional gain due to over-smoothing problem.

Therefore, an unaddressed need exists in the art to learn effect between distant nodes to classify aspect-level sentiment accurately.

SUMMARY OF THE DISCLOSURE

In certain aspects, the present disclosure relates to a system. In certain embodiments, the system includes a computing device, and the computing device has a processer and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to: receive a sentence having an aspect term and context, the aspect term having a classification label; convert the sentence into a dependency tree graph; calculate an attention matrix of the dependency tree graph based on one-hop attention between any two of a plurality of nodes in the dependency tree graph; calculate graph attention diffusion from multi-hop attention between any two of the plurality of nodes in the dependency tree graph based on the attention matrix; obtain updated embedding of the dependency tree graph using the graph diffusion attention; classify the aspect term based on the updated embedding of the dependency tree graph to obtain predicted classification of the aspect term; calculate loss function based on the predicted classification of the aspect term and the classification label of the aspect term; and adjust parameters of models in the computer executable code based on the loss function.

In certain embodiments, the computer executable code has an embedding module, attention module, and a dependency tree module. The embedding module, the attention module and the dependency tree module are configured to convert the sentence into the dependency tree graph by: embedding, by the embedding module, the aspect term to obtain an aspect term embedding X_t; converting, by the attention module, the aspect term embedding to aspect term feature H_α by: α=softmax (σ(W₂σ(W₁X_t^T))) and H_α=αX_t, wherein α∈^1×m, H_α∈^1×d, m is a positive integer indicating a number of embedding tokens of the aspect term, d is a positive integer indicating dimensions of embedding, X_t^T is the transposition of X_t, and σ denotes tanh activation function; embedding, by the embedding module, the context to obtain context embeddings X_c, each of the context embeddings corresponding to a word or a punctuation in the sentence that are not the aspect term; parsing, by the dependency tree module, the sentence into a dependency tree; and combining, by the dependency tree module, the aspect term feature H_α, the context embeddings X_c, and the dependency tree into the dependency tree graph, wherein each node of the dependency tree graph has a feature corresponding to one of the aspect term embedding and the context embeddings.

In certain embodiments, the computer executable code has a plurality of graph diffusion transformer (GDT) layers, and l-th GDT layer of the plurality of GDT layers is configured to calculate the attention matrix by: calculating an attention score s_i,j^(l)=σ₂ (v*σ₁(W_hh_i^(l)∥W_th_j^(l))) for nodes i and node j in the dependency tree graph, where W_h, W_t∈^d×d and v∈^2×d are learnable weights, h_i^(l) is a feature of node i at the l-th GDT layer, ∥ is a concatenation operation, σ₁ is a ReLU activation function, and σ₂ is a LeakyReLU activation function; obtaining attention score matrix S^(l) by:

$\begin{array}{cc}{S}^{\left(l\right)}=\{\begin{array}{cc}{s}_{i,j}^{\left(l\right)},& \mathrm{if}\mathrm{there}\mathrm{is}\mathrm{an}\mathrm{edge}\mathrm{between}i\mathrm{and}j\\ -\infty ,& \mathrm{otherwise}\end{array};& \end{array}$

and calculating the attention matrix A^(l) by: A^(l)=softmax(S^(l)).

In certain embodiments, the l-th GDT layer is configured to calculate the graph attention diffusion attention by: calculating diffusion attention matrix Ã^(l) by: Ã^(l)=Σ_hop=0^∞α(1−α)^hopA^hop(l), α∈(0, 1], where hop is a positive integer in a range of 2-12, and θ_hop is an attention decay factor; and calculating the graph attention diffusion H^(l+1) by: H^(l+1)=Ã^(l)H^(l), where H^(l) is input dependency tree graph embedding of the l-th layer.

In certain embodiments, the l-th GDT layer is configured to approximate the graph attention diffusion H^(l+1) by: Z^(k+1)=(1−α)A^(l)Z^k+αH^(l); Z⁽⁰⁾=H^(l); and H^(l+1)=Z^k, where k is an integer in a range of [0-K].

In certain embodiments, K is a positive integer in a range of 2-12, and l is a positive integer in a range of 2-24.

In certain embodiments, the l-th GDT layer is configured to obtain the updated embedding of the independency tree graph by: concatenating the graph attention diffusions H^(l+1) of a plurality of heads to obtain concatenated attention diffusion Ĥ^(l+1) by: Ĥ^(l+1)=Concat(h₁^(l+1), . . . , h_T^(l+1))W⁰, wherein each of h₁^(l+1), . . . , h_T^(l+1)) corresponds to one of the plurality of graph attention diffusions H^(l+1), W⁰=^Tdh×Tdh, T is a number of heads, d_h is hidden dimensions of each head, and d_h=d/T; performing {tilde over (H)}^l+1=Ĥ^(l+1)+Norm(H^(l)); and performing H^(l+1)=W₂ (σ(W₁Norm({tilde over (H)}^l+1)))+{tilde over (H)}^l+1, wherein W₁=∈^d×d and W₂=∈^d×d are trainable matrix, and a represents ReLU activation function, where H^(l+1) is the updated embedding of the independency tree graph.

In certain embodiments, the computer executable code comprises a classifier, and the classifier is configured to classify the aspect term by: ŷ=W₂σ(W₁Ĥ_t^T), where W₂=∈^C×dout and W₁=∈^dout×dh are learnable weight matrix, C is class number of the classification, σ is tanh activation function, Ĥ_t is aspect term embedding extracted from the updated embedding H^(l+1), Ĥ_t^T is transpose of Ĥ_t^T, and d_out is dimensions of H^(l+1).

In certain embodiments, the classifier is configured to calculate the loss function by: loss=−Σ_c=1^Cy log ŷ+λ∥θ∥², where λ is a coefficient for L2-regularization, θ are parameters that need to be regularized, and y is the label of the aspect term.

In certain embodiments, the computer executable code is further configured to: receive a query sentence and a query aspect term from the query sentence; convert the query sentence into a query dependency tree graph; calculate a query attention matrix of the query dependency tree graph based on one-hop attention between any two of a plurality of the nodes in the query dependency tree graph; calculate a query graph attention diffusion from the query attention matrix; obtain updated query embedding of the query dependency tree graph using the query graph attention diffusion; classify the query aspect term based on the updated query embedding of the query dependency tree graph to obtain predicted classification of the query aspect term; and label the query aspect term with the predicted classification.

In certain aspects, the present disclosure relates to a method. In certain embodiments, the method includes: receiving, by a computing device, a sentence having an aspect term and context, the aspect term having a classification label; converting, by the computing device, the sentence into a dependency tree graph; calculating, by the computing device, an attention matrix of the dependency tree graph based on one-hop attention between any two of the plurality of the nodes in the dependency tree graph; calculating, by the computing device, graph attention diffusion from multi-hop attention between any two of the plurality of nodes in the dependency tree graph based on the attention matrix; obtaining, by the computing device, updated embedding of the dependency tree graph using the graph attention diffusion; classifying, by the computing device, the aspect term based on the updated embedding of the dependency tree graph to obtain predicted classification of the aspect term; calculating, by the computing device, loss function based on the predicted classification of the aspect term and the classification label of the aspect term; and adjusting, by the computing device, parameters of models in the computer executable code.

In certain embodiments, the step of converting the sentence into the dependency tree graph includes: embedding the aspect term to obtain an aspect term embedding X_t; converting the aspect term embedding to aspect term feature H_α by: α=softmax (σ(W₂σ(W₁X_t^T))) (1) and H_α=αX_t (2), wherein α∈^1×m, H_α∈^1×d, m is a positive integer indicating a number of embedding tokens of the aspect term, d is a positive integer indicating dimensions of embedding, X_t^T is the transposition of X_t, and σ denotes tanh activation function; embedding the context to obtain context embeddings X_c, each of the context embeddings corresponding to a word or a punctuation in the sentence that are not the aspect term; parsing the sentence into a dependency tree; and combining the aspect term feature H_α, the context embeddings X_c, and the dependency tree into the dependency tree graph, wherein each node of the dependency tree graph has a feature corresponding to one of the aspect term embedding and the context embeddings.

In certain embodiments, the step of calculating the graph attention matrix includes: calculating an attention score s_i,j^(l)=σ₂ (v*σ₁ (W_hh_i^(l)∥W_th_j^(l)) for nodes i and node j in the dependency tree graph, wherein W_h, W_t ∈^d×d and v∈^2×d are learnable weights, h_i^(l) is a feature of node i at l-th graph diffusion attention (GDT) layer of a plurality of GDT layers, ∥ is a concatenation operation, σ₁ is a ReLU activation function, and σ₂ is a LeakyReLU activation function; obtaining attention score matrix S^(l) by:

$\begin{array}{cc}{S}^{\left(l\right)}=\{\begin{array}{cc}{s}_{i,j}^{\left(l\right)},& \mathrm{if}\mathrm{there}\mathrm{is}\mathrm{an}\mathrm{edge}\mathrm{between}i\mathrm{and}j\\ -\infty ,& \mathrm{otherwise}\end{array};& \end{array}$

calculating the attention matrix A^(l) by: A^(l)=softmax(S^(l)); calculating diffusion attention matrix Ã^(l) by: Ã^(l)=Σ_hop=0^∞α(1−α)^hopA^hop(l), α∈(0, 1], wherein hop is a positive integer in a range of 2-12, and θ_hop is an attention decay factor; and calculating the graph attention diffusion H^(l+1) by: H^(l+1)=Ã^(l) H^(l), wherein H^(l) is input dependency tree graph embedding of the l-th GDT layer.

In certain embodiments, the H^(l+1) is approximated by: Z^(k+1)=(1−α)A^(l) Z^k+αH^(l); Z⁽⁰⁾=H^(l); and H^(l+1)=Z^k, where k is an integer in a range of [0-K].

In certain embodiments, the step of obtaining the updated embedding of the independency tree graph includes: concatenating the graph attention diffusions H^(l+1) of a plurality of heads to obtain concatenated attention diffusion Ĥ^(l+1) by: Ĥ^(l+1)=Concat(h₁^(l+1), . . . , h_T^(l+1)) W⁰, where each of h₁^(l+1), . . . , h_T^(l+1) corresponds to one of the plurality of graph attention diffusions H^(l+1), W⁰=^Tdh×Tdh, T is a number of heads, d_h is hidden dimensions of each head, and d_h=d/T; performing {tilde over (H)}^l+1=Ĥ^(l+1)+Norm(H^(l)); and performing H^(l+1)=W₂ (σ(W₁Norm({tilde over (H)}^(l+1))))+Ĥ^(l+1), where W₁=∈^d×d and W₂=∈^d×d are trainable matrix, a represents ReLU activation function, and H^(l+1) is the updated embedding of the independency tree graph.

In certain embodiments, the step of classifying the aspect term includes: ŷ=W₂σ(W₁Ĥ_t^T), where W₂=∈^C×dout and W₁=∈^dout×dh are learnable weight matrix, C is class number of the classification, σ is tanh activation function, Ĥ_t is aspect term embedding extracted from the updated embedding H^(l+1), Ĥ_t^T is transpose of Ĥ_t, and d_out is dimensions of H^(l+1).

In certain embodiments, the loss function is calculated by: loss=−Σ_c=1^Cy log ŷ+λ∥θ∥², where λ is a coefficient for L2-regularization, θ are parameters that need to be regularized, and y is the label of the aspect term.

In certain embodiments, the method further includes: receiving a query sentence and a query aspect term from the query sentence; converting the query sentence into a query dependency tree graph; calculating a query attention matrix of the query dependency tree graph based on one-hop attention between any two of a plurality of the nodes in the query dependency tree; calculating a query graph attention diffusion from the query attention matrix; obtaining updated query embedding of the query dependency tree graph using the query graph attention diffusion; classifying the query aspect term based on the updated query embedding of the query dependency tree graph to obtain predicted classification of the query aspect term; and labeling the query aspect term with the predicted classification.

In certain aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. The computer executable code, when executed at a processor of a computing device, is configured to perform the method described above.

These and other aspects of the present disclosure will become apparent from following description of the preferred embodiment taken in conjunction with the following drawings and their captions, although variations and modifications therein may be affected without departing from the spirit and scope of the novel concepts of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate one or more embodiments of the disclosure and together with the written description, serve to explain the principles of the disclosure. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like elements of an embodiment.

FIG. 1 schematically depicts a dependency tree of a sentence according to certain embodiments of the present disclosure.

FIG. 2 schematically depicts an architecture of a GDT model according to certain embodiments of the present disclosure.

FIG. 3A schematically depicts a GDT system according to certain embodiments of the present disclosure.

FIG. 3B schematically depicts a GDT block according to certain embodiments of the present disclosure.

FIG. 4 schematically depicts a training process for a sentiment classification application according to certain embodiments of the present disclosure.

FIG. 5 schematically depicts a method for aspect level sentiment classification according to certain embodiments of the present disclosure.

FIG. 6, Table 1 shows datasets for comparing the model according to certain embodiments of the present disclosure and other related models.

FIG. 7, Table 2 shows comparison of the model according to certain embodiments of the present disclosure with various baselines.

FIG. 8A schematically depicts an aspect term-opinion pair, where the aspect term can be correctly predicted by the sentiment classification application according to certain embodiments of the present disclosure.

FIG. 8B schematically depicts another aspect term-opinion pair, where the aspect term can be correctly predicted by the sentiment classification application according to certain embodiments of the present disclosure.

FIG. 8C schematically depicts a further aspect term-opinion pair, where the aspect term can be correctly predicted by the sentiment classification application according to certain embodiments of the present disclosure.

FIG. 9 schematically depicts impact of number of layers on Restaurant dataset by GAT and GDT.

FIG. 10 schematically depicts attention distribution of GAT and GDT.

DETAILED DESCRIPTION

The present disclosure is more particularly described in the following examples that are intended as illustrative only since numerous modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the disclosure are now described in detail. Referring to the drawings, like numbers indicate like components throughout the views. As used in the description herein and throughout the claims that follow, the meaning of “a” “an”, and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise. Moreover, titles or subtitles may be used in the specification for the convenience of a reader, which shall have no influence on the scope of the present disclosure. Additionally, some terms used in this specification are more specifically defined below.

The terms used in this specification generally have their ordinary meanings in the art, within the context of the disclosure, and in the specific context where each term is used. Certain terms that are used to describe the disclosure are discussed below, or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the disclosure. It will be appreciated that same thing can be said in more than one way. Consequently, alternative language and synonyms may be used for any one or more of the terms discussed herein, nor is any special significance to be placed upon whether or not a term is elaborated or discussed herein. The use of examples anywhere in this specification including examples of any terms discussed herein is illustrative only, and in no way limits the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various embodiments given in this specification.

As used herein, the term “module” may refer to, be part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor.

The term “code”, as used herein, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, and/or objects. The term shared, as used above, means that some or all code from multiple modules may be executed using a single (shared) processor. In addition, some or all code from multiple modules may be stored by a single (shared) memory. The term group, as used above, means that some or all code from a single module may be executed using a group of processors. In addition, some or all code from a single module may be stored using a group of memories.

The term “interface”, as used herein, generally refers to a communication tool or means at a point of interaction between components for performing data communication between the components. Generally, an interface may be applicable at the level of both hardware and software, and may be uni-directional or bi-directional interface. Examples of physical hardware interface may include electrical connectors, buses, ports, cables, terminals, and other I/O devices or components. The components in communication with the interface may be, for example, multiple components or peripheral devices of a computer system.

Overview of the Model

In certain aspects, the present disclosure provides a graph diffusion transformer (GDT) model for aspect term sentiment classification. FIG. 2 schematically depicts an overall architecture of the GDT model according to certain embodiments of the present disclosure. As shown in FIG. 2, for a training input of term-sentence pair, all the words in the sentence except for the aspect term are considered as context words. The aspect term is “food,” and the sentence is “The food is great,” and the context is “The is great.” The aspect term is embedded using BERT 202, the term embeddings of the aspect term may have several embedding tokens, and then the self-attention 204 processes the term embeddings to form one term feature, which is a single vector representing the aspect term. The context is also embedded using the BERT 202 to form embeddings of the context words and optionally the punctuations of the sentence. As described above, the BERT 202 may use a token as an embedding unit, and one context word may correspond to several token embeddings. The token embeddings for each context word are further averaged to obtain the final context embeddings, where each word or optionally each punctuation in the context corresponds to one context embedding. The disclosure further uses a parser 206 to parse the sentence into dependency tree. The dependency tree, the term feature, and the context embeddings are combined to form the dependency tree graph 208, where the parser 206 provide the tree structure of the sentence, and the aspect feature and the context embedding provide features for the nodes in the dependency tree graph 208. Each node corresponds to one word or a punctuation in the sentence, and the features of the nodes are represented by embedding vectors such as h₁, h₂, h₃, and h₄. The dependency tree graph 208 is then processed by the graph diffusion transformer (GDT) 210. The GDT 210 updates the node embeddings of the dependency tree graph 208, and updated node embeddings include updated aspect term embedding (updated term feature). The GDT 210 has N number of GDT layers 2100 linked tandemly. The GDT layers 2100 have the same structure, but parameters of different GDT layers 2100 would be different after training. As shown in FIG. 2, each GDT layer 2100 include sequentially a multi-head attention diffusion 2102, a layer normalization 2104 and a feed forward 2106. The updated term feature is outputted from the last GDT layer 2100 to a 2-layer MLP 212. The 2-layer MLP 212 makes prediction based on the updated term feature. The prediction can be, for example a classification of positive, negative, and neutral. During training, the prediction for the aspect term is compared with the true label of the aspect term to calculate a loss function, and the loss function is used to adjust the parameters of the BERT 202, the self-attention 204, the GDT 210 and the 2-layer MLP 212. The training may be performed several times for the same aspect-sentence pair, and may be performed using different aspect-sentence pairs. When the model is well-trained, the well-trained model can be used for prediction of an aspect-sentence pair that does not have label/classification of the aspect term. During prediction, classification by the 2-layer MLP 212 is the result, and calculation of loss function may not be necessary. The prediction of the aspect term can then be used to label the aspect term-sentence pair. In certain embodiments, the aspect term-sentence pair for labeling and the aspect term-sentence pair for training are preferably in the same field. For example, both of the training pairs and prediction pairs may be customer comments on products in an ecommerce platform.

Encoder for Aspect Term and Context

BERT Encoder. The disclosure uses the pre-trained BERT model 202 as the encoder to get initial representations of the aspect term and the context words. First, the disclosure constructs the input as “[CLS]+Sentence+[SEP]+Term+[SEP]” and feeds the input into the BERT 202, where [CLS] is the class token placed at the beginning of the input sample, [SEP] is a separator, “Sentence” is the tokens of the sentence, and “Term” is the token of the aspect term from the “sentence.” Note that for simplicity, the disclosure considers the aspect term as one single word. Suppose a sentence consists of n words (thus there are n−1 context words) and the BERT output of the term word has m sub-tokens. Then, the outputs of sentence words from BERT are treated as the embedding of context words X_c ∈^(n−1)×dB. Similarly, term representation X_t ∈^m×dB is obtained. Here is a real number, and d_B is the dimension of the BERT output.

Self-attention layer. After obtaining the embedding of the aspect term X_t, the disclosure applies self-attention 204 to summarize the information carried by each sub-token of the aspect term and get a single feature representation as the term feature. In certain embodiments, the disclosure utilizes a two-layer Multi-Layer Perceptron (MLP) to compute the scores of the sub-tokens and get weighted sum over all sub-tokens. Each of the scores indicates importance of the corresponding sub-token, and the weighted sum refers to summation of the representations of the sub-tokens according to their importance. This is formulated as follows:

α=softmax(σ(W₂σ(W₁X_t^T)))  (1)

H _α =αX _t  (2)

where α∈^1×m, H_α is the weighted sum or the aspect term representation, H_α∈^1×dB, X_t^T is the transposition of X_t, and σ denotes tanh activation function. The bias vectors are not shown here for simplicity.

GDT Layer

The GDT model 210 is designed to allow each node to interact with its multi-hop neighbors in one single layer, and it takes the dependency tree structure into consideration when calculating the attention score between any pair of nodes. When the disclosure stacks multiple GDT layers 2100, it alleviates the over-smoothing problem that deeper GAT model has.

The computation involves in one GDT layer, the l-th GDT layer 2100, is as follows: the disclosure first computes one-hop attention score matrix and then according to the one-hop matrix, calculate the multi-hop attention matrix (diffusion attention matrix) and attentively update each node's feature. The disclosure uses H_α as the aspect term representation, H_c (i.e., X_c) as the context representation. The combination of the aspect term representation H_α and the context representation H_c are the node embedding or node representation H^(l).

One-hop attention matrix. In the layer l of the stacked GDT layers 210, the GDT layer l first uses the node embeddings H^(l) to calculate the attention score s_i,j^(l) between the node i and its one-hop neighbor j as:

s _i,j ^(l)=σ₂(v*σ₁(W_hh_i^(l)∥W_th_j^(l)))  (3)

where W_h, W_t ∈^d×d and v∈^2×d are learnable weights. h_i^(l) denotes the feature of node i at the layer l, which comes from the node embeddings H^(l). ∥ represents the concatenation operation. σ₁ and σ₂ are ReLU and LeakyReLU activation function respectively.

Then the disclosure obtains one-hop attention score matrix (or briefly attention score matrix) S^(l) as:

$\begin{array}{cc}{S}^{\left(l\right)}=\{\begin{array}{cc}{s}_{i,j}^{\left(l\right)},& \mathrm{if}\mathrm{there}\mathrm{is}\mathrm{an}\mathrm{edge}\mathrm{between}i\mathrm{and}j\\ -\infty ,& \mathrm{otherwise}\end{array}.& \end{array}$

Next, we perform row-wise softmax operation on S^(l) and get the one-hop attention matrix (or briefly attention matrix) A^(l):

A ^(l)=softmax(S^(l)).  (4)

Multi-hop attention matrix (or diffusion attention matrix). The multi-hop attention matrix Ã^(l) could be estimated by the one-hop attention matrix A^(l) as follows:

à ^(l)=Σ_i=0^∞α(1−α)ⁱA^i(l),α∈(0,1]  (5),

where α is the attention decay factor. A^i(l) is the i-th power of A^(l), representing i-hop attention score matrix. With Ã^(l), the disclosure can update node features or node embeddings as:

H ^(l+1) =Ã^(l)H^(l)  (6)

However, computing the power of A^(l) is very expensive and could be the bottleneck during training process. Thus, the disclosure approximates H^(l+1) by:

Z ^(k+1)−(1−α)A^(l)Z^k+αH^(l)  (7),

Z ⁽⁰⁾ =H ^(l)  (8), and

H ^(l+1) =Z^k  (9).

Here α is a predefined constant in a range of 0-0.5. In certain embodiments, α is in a range of 0.05 to 0.25. In certain embodiments, α is in a range of 0.1 to 0.2. In certain embodiments, α is 0.1 or 0.15. 0≤k≤K. K is the hop number, and is a positive integer in a range of 3-10. In certain embodiments, K is in a range of 4-8. In certain embodiments, K is 6. The values of α and K may vary according to the size and features of the knowledge graph 332. For example, assume that α is 0.1 and K is 6, then Z⁽⁰⁾=H^(l), Z⁽¹⁾=0.9A^(l) Z⁽⁰⁾+0.1Z⁽⁰⁾, Z⁽²⁾=0.9A^(l) Z⁽¹⁾+0.1Z⁽⁰⁾, Z⁽³⁾=0.9A^(l) Z⁽²⁾+0.1Z⁽⁰⁾, Z⁽⁴⁾=0.9A^(l) Z⁽³⁾+0.1Z⁽⁰⁾, Z⁽⁵⁾=0.9A^(l) Z⁽⁴⁾+0.1Z⁽⁰⁾, Z⁽⁶⁾=0.9A^(l) Z⁽⁵⁾+0.1Z⁽⁰⁾, and Z⁽⁶⁾ is the graph attention diffusion, which is the approximation of H^(l+1) or Ã^(l) H^(l). As described above, the calculation of the graph attention diffusion H^(l+1)can also be performed using the equations (5) and (6). However, because the Z⁽¹⁾, Z⁽²⁾, Z⁽³⁾, Z⁽⁴⁾, Z⁽⁵⁾, and Z⁽⁶⁾ are calculated recursively, the calculation is much faster than the calculation using the equations (7), (8) and (9).

The disclosure defines the feature aggregation and update process as:

H ^(l+1) =Attn_Diffusion(H^(l),)  (10),

where denotes the dependency tree graph.

Multi-head Attention Diffusion Layer. Instead of performing a single attention function, the disclosure applies multi-head attention mechanism that allows the model to jointly attend to information at different positions:

Ĥ ^(l+1)=Concat(h₁^(l+1), . . . , h_T^(l+1))W⁰  (11),

h _i ^(l+1)=Attn_Diffusion(h_i^(l),)  (12)

where W⁰=^Tdh×Tdh, T is the head number, d_h is the hidden dimension of each head. d_h and d are constrained by d_h=d/T, and h_i^(l+1) here for each head is equivalent to H^(l+1).

Layer Normalization and Feed Forward. Next, the disclosure applies the layer normalization operation on H^(l) and add the normalized feature to Ĥ^l+1 as the residual connection:

{tilde over (H)} ^l+1 =Ĥ ^l+1+Norm(H^(l))  (13),

where Norm denotes the layer normalization.

Then, instead of directly using activation function on H^(l) in GAT, the disclosure applies a learnable feed-forward layer and then add a residual connection as well:

H ^(l+1) =W ₂(σ(W₁Norm({tilde over (H)}^l+1)))+{tilde over (H)}^l+1  (14)

where W₁=∈^d×d and W₂=∈^d×d are trainable matrix and σ represents ReLU activation function.

Classifier. The disclosure extracts the aspect term node feature Ĥ_t ∈^d×d from Ĥ_out, which is the output of the last GDT layer. Then the disclosure feeds it into a two-layer MLP to calculate the final classification scores:

ŷ=W ₂σ(W₁Ĥ_t^T)  (15),

where W₂=∈^C×dout and W₁=∈^dout×dh denote the learnable weight matrix, C is the class number, which is 3 in certain embodiments of the present disclosure, σ refers to tanh activation function.

In certain embodiments, the disclosure uses cross entropy as the loss function:

loss=−Σ_c=1^Cy log ŷ+λ∥θ∥²  (16),

where λ is the coefficient for L2-regularization, θ denotes the parameters that need to be regularized, y is the true label, and ŷ is the predicted result.

Implementation of the Present Disclosure in a Computing Device

The present disclosure relates to computer systems. As depicted in the drawings, computer components may include physical hardware components, which are shown as solid line blocks, and virtual software components, which are shown as dashed line blocks. One of ordinary skill in the art would appreciate that, unless otherwise indicated, these computer components may be implemented in, but not limited to, the forms of software, firmware or hardware components, or a combination thereof.

The apparatuses, systems and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.

The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

FIG. 3A schematically depicts an aspect-level sentiment classification system according to certain embodiments of the present disclosure. As shown in FIG. 3A, the system 300 includes a computing device 310. In certain embodiments, the computing device 310 may be a server computer, a cluster, a cloud computer, a general-purpose computer, a headless computer, or a specialized computer, which provides note classification or knowledge graph completion services. The computing device 310 may include, without being limited to, a processor 312, a memory 314, and a storage device 316. In certain embodiments, the computing device 310 may include other hardware components and software components (not shown) to perform its corresponding tasks. Examples of these hardware and software components may include, but not limited to, other required memory, interfaces, buses, Input/Output (I/O) modules or devices, network interfaces, and peripheral devices.

The processor 312 may be a central processing unit (CPU) which is configured to control operation of the computing device 310. The processor 312 can execute an operating system (OS) or other applications of the computing device 310. In certain embodiments, the computing device 310 may have more than one CPU as the processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory 314 can be a volatile memory, such as the random-access memory (RAM), for storing the data and information during the operation of the computing device 310. In certain embodiments, the memory 314 may be a volatile memory array. In certain embodiments, the computing device 310 may run on more than one memory 314. The storage device 316 is a non-volatile data storage media for storing the OS (not shown) and other applications of the computing device 310. Examples of the storage device 316 may include non-volatile memory such as flash memory, memory cards, USB drives, hard drives, floppy disks, optical drives, solid-state drive, or any other types of data storage devices. In certain embodiments, the computing device 310 may have multiple storage devices 316, which may be identical storage devices or different types of storage devices, and the applications of the computing device 310 may be stored in one or more of the storage devices 316 of the computing device 310.

In this embodiment, the processor 312, the memory 314, and the storage device 316 are component of the computing device 310, such as a server computing device. In other embodiments, the computing device 310 may be a distributed computing device and the processor 312, the memory 314, and the storage device 316 are shared resources from multiple computing devices in a pre-defined area.

The storage device 316 includes, among other things, a sentiment classification application 318 and opinions 334. The sentiment classification application 318 is configured to train its model structure using aspect-level sentiment labels of the opinion 332, and make predictions for the opinions that do not have aspect-level sentiment labels. The opinions 334 is optional for the computing device 310, as long as the opinions stored in other devices is accessible to the sentiment classification application 318.

As shown in FIG. 3 A, the sentiment classification application 318 includes an embedding module 320, an attention module 322, a dependency tree module 324, graph diffusion transformer (GDT) layers 326, a classifier 328, a function module 330, and an interface 332. In certain embodiments, the sentiment classification application 318 may include other applications or modules necessary for the operation of the sentiment classification application 318. It should be noted that the modules 320-332 are each implemented by computer executable codes or instructions, or data table or databases, or a combination of hardware and software, which collectively forms one application. In certain embodiments, each of the modules may further include sub-modules. Alternatively, some of the modules may be combined as one stack. In other embodiments, certain modules may be implemented as a circuit instead of executable code. In certain embodiments, the modules can also be collectively named a model, which can be trained using training data, and after well trained, can be used to make a prediction.

The embedding module 320 is configured to embed an aspect term and context of an opinion from the opinions 334 to obtain aspect term embedding X_t and context embedding X_c, send the aspect term embedding X_t to the attention module 322, and send the context embedding X_c to the dependency tree module 324. In certain embodiments, the opinion is a sentence, and the opinion includes the aspect term, which is also a word in the sentence. Before feeding into the embedding encoder, the embedding module 320 may prepare the sentence in the form of “[CLS]+Sentence+[SEP]+Term+[SEP].” After embedding by the embedding module 320, the outputted embedding includes the term representation X_t and the context words embedding X_c. The term representation X_t may have embeddings for one or a few sub-tokens. Each vector in the vector matrix X_c corresponds to one word or a punctuation in the context. When a context word is separated to a few tokens, the token embedding is further averaged, such that each context word only has one corresponding vector in the context words embedding X_c. The embedding may be performed, for example, by deep bidirectional transformers (BERT), robustly optimized BERT pretraining approach (RoBERTa), or Word2vec.

The attention module 322 is configured to, upon receiving the aspect term embedding X_t from the embedding module 320, perform a two-layer MLP on the aspect term embedding to obtain the term feature H_α, and send the term feature H_α to the dependency tree module 324. In certain embodiments, the attention module 322 performs the two-layer MLP using the equations (1) and (2). In certain embodiments, the aspect term may be one or two words, and one aspect term word may correspond to one or multiple tokens. By operation of the attention module 322, the one or more embedding tokens of the aspect term X_t is converted to the one embedding-term feature H_α. As a result, the dimension of the term feature H_α is the same as the dimension of each context word embedding in the context words embedding X_c.

The sentence is available to the dependency tree module 324, and the dependency tree module 324 is configured to parse the sentence into a dependency tree. The dependency tree module 324 is further configured to, upon receiving the aspect term feature H_α from the attention module 322 and the context embedding X_c from the embedding module 320, combine the dependency tree, the term feature and the context embedding into a dependency tree graph, and send the dependency tree graph to the GDT layers 326. Referring back to FIG. 2, the dependency tree graph has each word (and optionally one or more punctuations) of the sentence as one node of the graph, each node is characterized by the aspect term feature H_α or the context embedding X_c, and certain nodes are connected according to the parsed dependency tree. The dependency tree module 324 may include any parsing methods to parse the sentence into the dependency tree, such as Stanford CoreNLP, Berkeley Neural Parser, and Stanza Parser.

The GDT layers 326 is configured to, upon receiving the dependency tree graph from the dependency tree module 324, perform graph diffusion to update node embeddings in the dependency tree graph, such that the updated nodes are available to the classifier 328. The GDT layers 326 may include one or multiple GDT layers that have the same layer or block structure, but each layer or block has its specific parameters. Assume there is a total number of L GDT layers, where L is a positive integer, the disclosure defines the L GDT layers as layer 0, layer 1, . . . , layer l(or the l-th layer), . . . , layer (L−1), where l is from 0 to L−1. The input of layer 0 is the initial node embeddings of the dependency tree graph, that is, H_α and X_c. The combination of H_α and X_C is the initial embedding H⁽⁰⁾ for the layer 0. The output of layer 0 is H^(l). Accordingly, the input and output for layer l are respectively H^(l) and H^(l+1). FIG. 3B schematically depicts one GDT layer 3260 according to certain embodiments of the present disclosure. As shown in FIG. 3B, the GDT layer 3260 includes a multi-head attention diffusion module 3261, a first layer normalization module 3262, a first addition module 3263, a second layer normalization module 3264, a feed forward module 3265, and a second addition module 3266.

At the start of a training of the sentiment classification application 318, the node embedding H⁽⁰⁾ is available for both the multi-head attention diffusion module 3261 and the first layer normalization module 3262 of the first GDT layer 3260 (layer 0). After operation of the l-th GDT block, the outputted node embedding for that block, i.e., H^(l+1), is available to both the multi-head attention diffusion module 3261 and the first layer normalization module 3262 of the next GDT layer. When the current GDT layer is the last GDT layer, the outputted node embedding is provided to the classifier 328.

The multi-head attention diffusion module 3261 is configured to, upon receiving the node embedding H^(l), compute graph attention diffusion head_i for each of T heads, concatenate the graph attention diffusions for all the heads to obtain concatenated attention diffusion Ĥ^(l+1) using the equation (11), and send the concatenated attention diffusion Ĥ^(l+1) to the first addition module 3263. The attention diffusion for head i is calculated by equation (12), which can be calculated from the equations (3), (4), (5), and (6) using the node embedding H^(l) or approximated using equations (3), (4), (7), (8) and (9).

The first layer normalization module 3262 is configured to, upon receiving the inputted node embeddings H^(l) at the l-th layer, perform layer normalization on the inputted node embeddings H^(l) to obtain first normalized embedding Norm(H^(l)), and send the first normalized embedding Norm(H^(l)) to the first addition module 3263. In certain embodiments, the first layer normalization module 3262 may also perform its function before the operation of the multi-head attention diffusion module 3261.

The first addition module 3263 is configured to, upon receiving the concatenated attention diffusion Ĥ^(l+1) from the multi-head attention diffusion module 3261 and the first normalized embedding Norm(H^(l)) from the first layer normalization module 3262, add the concatenated attention diffusion Ĥ^(l+1) to the normalized embedding Norm(H^(l)) to obtain embedding with first addition {tilde over (H)}^l+1 (equation (13)), and send the embedding with the first addition {tilde over (H)}^l+1 to the second layer normalization module 3264 and the second addition module 3266.

The second layer normalization module 3264 is configured to, upon receiving the embedding with the first addition {tilde over (H)}^l+1, normalize the embedding to obtain second normalized embedding Norm({tilde over (H)}^l+1), and send the second normalized embedding Norm({tilde over (H)}^l+1) to the feed forward module 3265.

The feed forward module 3265 is configured to, upon receiving the second normalized embedding Norm({tilde over (H)}^l+1), perform feed forward to obtain feed forward embedding W₂ (σ(W₁Norm({tilde over (H)}^l+1))), and send the feed forward embedding to the second addition module 3226.

The second addition module 3266 is configured to, upon receiving the embedding with the first addition {tilde over (H)}^l+1 from the first addition module 3263 and the feed forward embedding W₂ (σ(W₁Norm({tilde over (H)}^l+1))) from the feed forward module 3265, perform an addition of the two to obtain the updated node embedding H^(l+1) by equation (14), such that the updated node embedding H^(l+1) is available to the next GDT layer, or available to the classifier 328 when the current layer is the last GDT layer. Kindly note the operations of the second layer normalization module 3264m the feed forward module 3265 and the second addition module 3266 are preferably performed together according to the equation (14).

The classifier 328 is configured to, upon receiving the updated node embedding H^(l+1) from the second addition module 3266 of the last GDT layer (here the last GDT layer is layer l), extract term node feature Ĥ_t from the updated node embedding H^(l+1), feed the term node feature Ĥ_t to a two-layer MLP to calculate the final classification score ŷ of the term node using the equation (15), calculate a loss function using the equation (16), and adjust model parameters based on the loss function. The model parameters include parameters for the embedding module 320, the attention module 322, the GDT layers 326, and the classifier 328. During prediction, the classification score f is used to give the prediction of the sentiment classification of the aspect term, and there is no need to calculate the loss function and adjust parameters. In certain embodiments, the opinions 334 for training and prediction may be different but related datasets. For example, the training opinions may be customer comments with labeled aspect terms, while the opinions for prediction may be customer comments with aspect terms, but the aspect terms are not labeled with sentiment classification.

The function module 330 is configured to, when the classification of an aspect term is determined for an aspect term-sentence pair, add the classification label of the aspect term to the aspect term-sentence pair. When a large number of aspect term-pairs are labeled, the function module 330 is further configured to present the labeled opinions to the customers. The classification, for example, can be “positive,” “negative,” or “neutral.” In certain embodiments, the function module 330 may perform function of presentation to the customers by recommending a product to the customer when an aspect term of an opinion about a product is positive, and the customer is searching product using the aspect term. In an example, opinions or comments from customers on products are available, and aspect terms of the products are predefined. The predefined terms may by rule-based. For example, “quality” and “price” may be defined as a rule, and the application may find the “quality” and “price” terms in an opinion and define them as aspect terms of the opinion. After receiving the opinions with labeled aspect terms, the sentiment classification application 318 is configured to classify the aspect terms into positive, negative, or neutral, and present the results to customers. When an opinion on a product include two aspect terms, one aspect term such as “quality” may be positive (high quality), and the other aspect term “price” may be negative (expensive). When a new customer review the opinion and she cares more about the “quality” aspect, she may choose to buy the product because the classification of the “quality” is positive; if she cares more about the “price” aspect, she may choose not to buy the product because the classification of the “price” is negative. By providing the opinions based on their aspect terms, the sentiment classification application 318 provides a fine-grained result for the customers to make decisions.

In certain embodiments, the function module 328 is configured to perform the above function automatically or in a predefined time interval, or when trigged by an update of the opinions 334. For example, after the update of the opinions 334, the function module 328 would look for more linked relations between products and customers, and the function module 328 would subsequently push the products to the corresponding customers when the updated relations between the customers and the products are interested in.

The interface 330 is configured to provide an interface for an administrator of the sentiment classification application 318 to train the embedding module 320, the attention module 322, the GDT layers 326, the classifier 328, and optionally the dependency tree module 324 and the function module 330, and adjust model parameters, or is configured to provide a user interface for the administrator to use the sentiment classification application 318 to obtain and use the opinions 334 for certain functions.

FIG. 4 schematically depicts a training process for sentiment classification application according to certain embodiments of the present disclosure. In certain embodiments, the training process is implemented by the computing device 310 shown in FIG. 3A. It should be particularly noted that, unless otherwise stated in the present disclosure, the steps of the training process or method may be arranged in a different sequential order, and are thus not limited to the sequential order as shown in FIG. 4.

As shown in FIG. 4, at procedure 402, an aspect term-sentence pair is provided, the embedding module 320 embeds the aspect term to aspect term embeddings, and sends the aspect term embeddings to the attention module 322.

At procedure 404, the attention module 322 processes the aspect term embeddings to obtain term feature, and sends the term feature to the dependency tree module 324. The aspect term embeddings may include embeddings of one or multiple tokens, while the obtained term feature is one embedding has the dimension of d_B.

At procedure 406, the embedding module 320 embeds the sentence without the aspect term to obtain context embeddings, and sends the context embeddings to the dependency tree module 324. When a context word has multiple embedding tokens, the embedding tokens are averaged, such that each embedding in the obtained context embeddings corresponds to one word in the sentence context. In certain embodiments, punctuation in the sentence also has a corresponding embedding. In certain embodiments, the procedures 402 and 406 are performed together, for example, using the input “[CLS]+Sentence+[SEP]+Term+[SEP].”

At procedure 408, the dependency tree module 324 parses the sentence to obtain dependency tree of the sentence, each word and optionally the punctuation is a unit or a node in the dependency tree. Kindly note that the procedures 402-404, the procedure 406, and the procedure 408 may be performed in parallel.

At procedure 410, the dependency tree module 324, upon parsing the sentence and upon receiving the term feature and the context embedding, forms a dependency tree graph. The feature of each node in the dependency tree graph is either the aspect feature corresponding to the aspect term, or embedding of one word in the context of the sentence, or optionally an embedding of a punctuation of the sentence. When two words are linked in the dependency tree, the dependency tree graph will define it as an edge. In other words, the dependency tree provides the tree structure, and the term feature and context embeddings provide characters or features of the nodes. After obtaining the dependency tree graph, the dependency tree module 324 sends the dependency tree graph to the first GDT layer of the GDT layer 326.

At procedure 412, the GDT layers 326 processes the dependency tree graph in sequence. For each of the GDT layer 3260, for example layer l, the GDT layer performs multi-head attention to update the node embeddings. With the input H^l (which is H⁰ for the first GDT layer), the multi-head attention diffusion module 3261 calculates one-hop attention score s_i,j^(l) between the node i and the node j using the equation

$\begin{array}{cc}{s}_{i,j}^{\left(l\right)}={\sigma }_2\left(v*{\sigma }_1\left(W_h{h}_i^{\left(l\right)}W_t{h}_j^{\left(l\right)}\right)\right),& \left(3\right)\end{array}$

where h_i^(l) and h_i^(l) denote the feature or embeddings of node i and j at the layer l, respectively. The h_i^(l) and h_i^(l) come from the node embeddings H^l. The multi-head attention diffusion module 3261 calculates one-hop attention score between all the nodes, and obtains the one-hop attention matrix S^(l) as:

$\begin{array}{cc}{S}^{\left(l\right)}=\{\begin{array}{cc}{s}_{i,j}^{\left(l\right)},& \mathrm{if}\mathrm{there}\mathrm{is}\mathrm{an}\mathrm{edge}\mathrm{between}i\mathrm{and}j\\ -\infty ,& \mathrm{otherwise}\end{array}.& \end{array}$

At procedure 414, the multi-head attention diffusion module 3261 performs softmax on the one-hop attention score matrix S^(l) to obtain the one-hop attention matrix A^(l) by: A^(l)=softmax(S^(l)) (4).

At procedure 416, the multi-head attention diffusion module 3261 calculates multi-head attention matrix (also named diffusion attention matrix) Ã^(l) based on the one-hop attention matrix A^(l) by: Ã^(l)=Σ_i=0^∞α(1−α)ⁱA^i(l), α∈(0, 1] (5), where a is the attention decay factor. A^i(l) is the i-th power of A^(l), and represents i-hop attention matrix.

At procedure 418, the multi-head attention diffusion module 3261 calculates graph attention diffusion H^(l+1) by: H^(l+1)=Ã^(l) H^l(6).

In certain embodiments, to increase the calculation speed, the graph attention diffusion H^(l+1) is approximately calculated by:

Z ^(k+1)=(1−α)A^(l)Z^k+αH^l  (7),

Z ⁽⁰⁾ =H ^l  (8), and

H ^(l+1) =Z^k  (9).

In other words, the calculation by the equations (7), (8) and (9) can be used to replace the procedures 416 and 418, or to replace the calculations of the equations (5) and (6). As described above, α is a predefined constant in a range of 0-0.5, preferably in a range of 0.05 to 0.25, and could be for example 0.1 or 0.15. 0≤k≤K, and K is the hop number. K is a positive integer in a range of 3-10. Because Z^(k) can be calculated recursively, the calculation is much faster than the calculation using the equations (5) and (6).

At procedure 420, the procedures 412-418 are performed for each head, and the graph attention diffusion for each head are concatenated to obtain the concatenated graph attention diffusions by: Ĥ^(l+1)=Concat(h₁^(l+1), . . . , h_T^(l+1))W⁰ (11). The h_i^(l+1) in equation (11) is defined as: h_i^(l+1)+=Attn_Diffusion(h_i^(l), )(12). After obtaining the concatenated graph attention diffusion Ĥ^(l+1), the multi-head attention diffusion module 3261 further sends the concatenated graph attention diffusion to the first addition module 3263.

At procedure 422, the first layer normalization module 3262 performs layer normalization on the input node embedding H^l to obtain normalized embedding NormH^l, and sends the normalized embedding to the first addition module 3263.

At procedure 424, upon receiving the concatenated graph attention diffusion from the multi-head attention diffusion module 3261 and the normalized embedding Norm(H^(l)) from the first layer normalization module 3262, the first addition module 3263 adds the concatenated graph attention diffusion Ĥ^(l+1) to the normalized embedding of the previous GDT layer H^(l) or from the normalized initial embedding H⁰ (if the current GDT layer is the first GDT layer) to obtain added attention diffusion {tilde over (H)}^l+1 by: {tilde over (H)}^l+1=Ĥ^l+1+Norm(H^(l)) (13).

At procedure 426, upon receiving the added attention diffusion, the second layer normalization module 3264 performs layer normalization on the added graph attention diffusion to obtain normalized attention diffusion: Norm({tilde over (H)}^l+1), and sends the normalized attention diffusion to the feed forward module 3265.

At procedure 428, upon receiving the normalized attention diffusion from the second layer normalization module 3264, the feed forward module 3265 performs feed forward on the normalized attention diffusion to obtain the feed forward attention, and sends the feed forward attention to the second addition module 3266. The feed forward attention is W₂(σ(W₁Norm({tilde over (H)}^l+1))).

At procedure 430, upon receiving the feed forward attention from the feed forward module 3265, the second addition module 3266 adds the added attention diffusion to the feed forward attention, to obtain the updated embedding H^(l+1) that is: H^(l+1)=W₂ (σ(W₁Norm({tilde over (H)}^l+1)))+{tilde over (H)}^l+1 (14). After obtaining the updated embedding H^(l+1), the second addition module 3266 sends the updated embedding to the next GDT layer. In certain embodiments, the feed forward attention has a two-layer MLP network. When the layer l is the last GDT layer (or the last GDT block), then the final output of the GDT layers 326 is embedding H^(l+1).

At procedure 432, upon receiving the output embedding H^(l+1) from the last GDT layer 3260, the classifier 328 calculates a classification score for the aspect term embedding by: ŷ=W₂σ(W₁Ĥ_t^T) (15), where Ĥ_t is the embedding of the aspect term that can be extracted from the node embedding matrix H^(l+1).

At procedure 434, the classifier further calculates a loss function based on the classification score and the ground true label of the aspect term by loss=−Σ_c=1^Cy log ŷ+λθ∥² (16), and adjusts parameters of the sentiment classification application 318 based on the loss.

The whole training process may be performed iteratively using multiple training aspect term-sentence pairs, until the training is repeated for a predetermined number of times, or until the model parameters converge. Each aspect term-sentence pair may be used once or multiple times in the training.

FIG. 5 schematically depicts method 500 for aspect-term classification according to certain embodiments of the present disclosure, after the sentiment classification application 318 is well-trained. In certain embodiments, the method is implemented by the computing device 310 shown in FIG. 3A, specifically the function module 330. It should be particularly noted that, unless otherwise stated in the present disclosure, the steps of the prediction process or method may be arranged in a different sequential order, and are thus not limited to the sequential order as shown in FIG. 5. Kindly note that the training data of the Sentiment classification application 318 may be the same knowledge graph 332.

As shown in FIG. 5, at procedure 502, the function module 330 provides aspect term-sentence pairs. The sentences in the term-sentence pairs may be customer comments on an ecommerce platform, and the aspect terms may be extracted from the customer comments using a rule, such as defining price and quality of the products as the aspect terms.

At procedure 504, for each aspect term-sentence pair, the function module 330 calls the embedding module 320, the attention module 322, the dependency tree module 324, the GDT layers 326, and the classifier 328 to perform the procedures 402-432, so as to obtain the classification score of the aspect term in the corresponding sentence. There are a large number of aspect term-sentence pairs, and there are a large number of corresponding classification scores.

At procedure 506, the function module 330 ranks the classification scores, and labels the aspect terms that have the high classification scores. For example, the function module 330 may label the aspect terms whose classification scores are at a predetermined percentage of the rank, such as top 10% of the rank, or top 20% of the rank. In another example, the function module 330 may set a threshold classification score, and when an aspect term has a classification score that is greater than the threshold classification score, the classification is regarded as true and the function module 330 labels the aspect term with its predicted classification.

At procedure 508, the function module 330 adds the aspect term labels, the sentences, and the related products as a knowledge graph or database, and presents the knowledge graph or database to the customers.

At procedure 510, when the customers search or review the products, he can choose the product based on the product's aspect term labels, and the function module 330 may be, upon receiving an instruction from a customer that she is interested, help the customer to place an order of the product.

Experiments

Data Sets.

We evaluate our model on three datasets: Restaurant and Laptop reviews from SemEval 2014 Task 4 and ACL 14 Twitter dataset. We remove several examples with “conflict” labels in the reviews. The statistics of these datasets are listed in FIG. 6, Table 1. We obtain dependency trees using the Stanford CoreNLP (Manning et al., The Stanford CoreNLP natural language processing toolkit, Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, 2014, 55-60) for our experiments.

Baselines. We compare our model with following baseline models:

1. BERT-SPC (Song et al., Attentional encoder network for targeted sentiment classification, arXiv:1902.09314, 2019, which is incorporated herein by reference in its entirety) feeds the sentence and term pair into the BERT model and the BERT outputs are used for prediction.

2. AEN-BERT (Song et al., 2019 as above) uses BERT as the encoder and employs several attention layers.

3. TD-GAT-BERT (Huang and Carley, Syntax-aware aspect level sentiment classification with graph attention networks, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, 5472-5480) utilizes GAT on the dependency tree to propagate features from the syntactic context.

4. R-GAT+BERT (Want et al., Relational graph attention network for aspect-based sentiment analysis, arXiv:2004.12362, 2020) reshapes and prunes the dependency parsing tree to an aspect-oriented tree rooted at the aspect term, and then employs Relational GAT to encode the new tree for sentiment predictions.

We only choose strong baselines with BERT model as the encoder for fair comparison. Rietzler et al. (Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment, 2019) works on the same task with millions of extra data thus it's not comparable with our model. Zhang et al. (Aspect-based sentiment classification with aspect specific graph convolutional networks, EMNLP-IJCNLP, 2019, 4560-4570) and Sun et al. (Aspect-level sentiment analysis via convolution over dependency tree, EMNLP-IJCNLP, 2019, 5683-5692) both employ GNN over dependency trees to learn the syntax-aware representation of the aspect term, but they didn't use BERT as the encoder. Therefore, we implement our own GCN/GAT over dependency trees models based on BERT encoder as our baselines.

We implement our own BERT-Single by directly applying a classifier on top of BERT-base encoder. BERT+GAT and BERT+GCN employs GAT and GCN respectively on dependency trees based on the BERT-base encoder. The best results of the GAT and GCN models come from two or three layers, depending on different datasets. The implementations of GAT and GCN come from Deep Graph Library (DGL) (Want et al., Deep graph library: Towards efficient and scalable deep learning on graphs, 2019). BERT+GDT is our proposed GDT model with the BERT-base model as the encoder. Roberta+GDT replaces the encoder with the Roberta base model.

Parameter Setting. During training, we set the learning rate to 10⁻⁵. We set the batch size to 4. The dimension of BERT output d_B is 1536 (we take the concatenation of last two layers outputs from BERT). The dimension of the GDT model on dependency trees d is 256 or 512 according to different datasets. We use 4 or 8 heads during multi-head attention. We apply dropout (Srivastava, et al., Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 2014, 15(1):1929-1958) and L2 regularization. The coefficient rate λ of L2 is 10⁻⁶.

Experimental Results.

We present results of our GDT model in two aspects: classification performance and qualitative case study.

Classification. FIG. 7, Table 2 shows comparisons of the GDT model with other baselines in terms of classification accuracy and Macro-F1. From this table, we observe that: the GDT model achieves the best results on “Restaurant” and “Laptop” datasets and shows competitive results on the “Twitter” dataset as well. Applying Roberta-base model as the encoder in our model further boosts the performance on three datasets.

Specifically, both BERT+GCN and BERT+GAT outperform BERT-Single, which proves the benefit of using syntactic dependency tree information. BERT+GCN is worse than BERT+GAT, which shows that attentively aggregating information learns better node representations than treating each node equally.

The performance gain of the GDT model over the GAT model on Twitter domain is not as significant as Restaurant and Laptop domains. After checking into the data, the reason may be that the sentences in Twitter dataset are more colloquial and lack of grammatical norms, which is more likely to cause dependency parsing errors. This also explains the performance drop of BERT+GCN and BERT+GAT, compared to BERT-Single. Although our GDT model also operates on dependency parsing trees, it sees more context than GAT in single layer, thus more robust to parsing errors.

Qualitative case Study. The long distance between the aspect word and opinion context could be caused by parsing errors or original complex dependency structures of sentences. To show the efficacy of the GDT model on dealing with this problem, we demonstrate three examples. Specifically, FIG. 1 and FIG. 8A show long-hop examples due to parsing errors. FIG. 8B is a long-hop example caused by its inherent complex dependency structure. Aspect terms are “touch pad” in FIG. 8A and “Mac OS” in FIG. 8B, and the aspect terms are italic in the figures. These sentences are predicted correctly by the GDT model but wrongly by BERT+GAT.

The parsing error in FIG. 1 results in four hops between the aspect term “soup” and the opinion context “not tasty.” In the FIG. 8A example, there are six hops between the term “touch pad” and the opinion context “fine to work.” The FIG. 8B example has complex syntactic structure leading to a five-hop distance between “Mac OS” and “easily picked up.” All three examples are predicted wrong by the GAT model, because the shallow GAT model cannot learn the interactions between nodes so far away from each other. However, the GDT model could effectively aggregate information from these important opinion context in one layer and make the right prediction.

In addition, we also found that GDT shows better performance when dealing with examples that require global understanding of the whole sentence. FIG. 8C demonstrates a specific example. The sentiment label of the term “food” is neutral. In order to predict it correctly, the model needs to access the two phrases, “extra minutes” and “made up for it,” and then fuse the information from them. The GAT model classifies the sentiment as negative and the GDT model correctly predicts it as neutral. The reason is that, compared with the GAT model, our GDT model could make the aspect term see the two important context phrases in a single layer and effectively integrate their messages.

Model Analysis

We further analyzed our GDT model from two perspectives: number of layers analysis and attention distribution analysis.

Number of Layers Analysis. We explore the effect of the layer number on our model and GAT respectively. We vary the layer number from one to six and measure the corresponding accuracy on “Restaurant” dataset. The result is illustrated in FIG. 9. Similar results are found on “Laptop” and “Twitter” dataset. From FIG. 9, we can see that the GAT model achieves the best performance at two layers and then the overall accuracy goes down as the layer increases. The GDT model shows better performance with deeper layers and obtains the best result at six layers. This comparison shows that the GDT model is more robust to the increase of layers and could alleviate the over-smoothing problem to a certain extent. Besides, in each layer, the GDT model performs better than the GAT model, which further indicates that aggregating information from more context in one layer is more helpful to this task.

Attention Distribution Analysis. We also study the attention weights distributions of the GAT and GDT models. In this aspect-level sentiment classification task, in order to do the correct prediction, it's essential for an aspect term to locate the right opinion context. If the learnt attention weights distribution is close to uniform distribution, i.e., each node treats its neighbors with almost equal weights, then it's hard for the node to differentiate the truly important context words. Therefore, for the GAT and GDT model, we respectively measure its attention weights distribution difference to the uniform distribution on the “Restaurant” dataset. We use a discrepancy metric proposed in Shanthamallu, A regularized attention mechanism for graph attention networks, 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, 3372-3376, to calculate the difference:

$\begin{array}{cc}{d}_i=\frac{A_{i,:}-U_i}{\mathrm{degree}\left(v_i\right)},& \left(17\right)\end{array}$

where A_i,: represents the importance of each node in approximating the feature for node v_i, U_i is the uniform distribution score for node v_i. d_i measures the non-uniformity in the learned attention weights, the lower discrepancy value is, the stronger the uniformity of the learnt attention weight is.

The result is illustrated in FIG. 10. For fair comparison, we select the attention weights from the first layer's outputs in both models. We can see that the GAT model has more edges with attention weights difference close to zero, meaning that the GAT model learns more uniformly distributed attention weights compared to the GDT model. The result validates the statement that the GDT model is more capable to discriminate valuable context words and make better predictions.

In summary, the present disclosure provides a GDT model for the aspect-level sentiment classification task. We first encode the aspect term and context words by pre-trained BERT to capture the interaction between them. Then in order to handle the situations where the aspect terms are far away from opinion context, we design a GDT model on top of dependency trees to integrate multi-hop neighbor's information according to the dependency structure in one layer. Our extensive experiments show that our model achieves the new state-of-the-art results on the SemEval datasets and shows comparable results on the ACL 14 Twitter dataset.

The foregoing description of the exemplary embodiments of the disclosure has been presented only for the purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching.

The embodiments were chosen and described in order to explain the principles of the disclosure and their practical application so as to enable others skilled in the art to utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure pertains without departing from its spirit and scope. Accordingly, the scope of the present disclosure is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

Claims

1. A system comprising a computing device, the computing device comprising a processer and a storage device storing computer executable code, wherein the computer executable code, when executed at the processor, is configured to: receive a sentence having an aspect term and context, the aspect term having a classification label;convert the sentence into a dependency tree graph;calculate an attention matrix of the dependency tree graph based on one-hop attention between any two of a plurality of nodes in the dependency tree graph;calculate graph attention diffusion from multi-hop attention between any two of the plurality of nodes in the dependency tree graph based on the attention matrix;obtain updated embedding of the dependency tree graph using the graph diffusion attention;classify the aspect term based on the updated embedding of the dependency tree graph to obtain predicted classification of the aspect term;calculate loss function based on the predicted classification of the aspect term and the classification label of the aspect term; andadjust parameters of models in the computer executable code based on the loss function.

2. The system of claim 1, wherein the computer executable code comprises an embedding module, attention module, and a dependency tree module, and the embedding module, the attention module and the dependency tree module are configured to convert the sentence into the dependency tree graph by: embedding, by the embedding module, the aspect term to obtain an aspect term embedding Xt;converting, by the attention module, the aspect term embedding to aspect term feature Hα by: α=softmax (σ(W2σ(W1XtT))) and Hα=αXt, wherein α∈1×m, Hα∈1×d, m is a positive integer indicating a number of embedding tokens of the aspect term, d is a positive integer indicating dimensions of embedding, XtT is the transposition of Xt, and σ denotes tanh activation function;embedding, by the embedding module, the context to obtain context embeddings Xc, each of the context embeddings corresponding to a word or a punctuation in the sentence that are not the aspect term;parsing, by the dependency tree module, the sentence into a dependency tree; andcombining, by the dependency tree module, the aspect term feature Hα, the context embeddings Xc, and the dependency tree into the dependency tree graph, wherein each node of the dependency tree graph has a feature corresponding to one of the aspect term embedding and the context embeddings.

3. The system of claim 1, wherein the computer executable code comprises a plurality of graph diffusion transformer (GDT) layers, and l-th GDT layer of the plurality of GDT layers is configured to calculate the attention matrix by: calculating an attention score si,j(l)=σ2 (v*σ1 (Whhi(l)∥Wthj(l)) for nodes i and node j in the dependency tree graph, wherein Wh, Wt ∈d×d and v∈2×d are learnable weights, hi(l) is a feature of node i at the l-th GDT layer, ∥ is a concatenation operation, σ1 is a ReLU activation function, and σ2 is a LeakyReLU activation function;obtaining attention score matrix S(l) by:

4. The system of claim 3, wherein the l-th GDT layer is configured to calculate the graph attention diffusion attention by: calculating diffusion attention matrix Ã(l) by: Ã(l)=Σhop=0∞α(1−α)hopAhop(l), α∈(0, 1], wherein hop is a positive integer in a range of 2-12, and θhop is an attention decay factor; andcalculating the graph attention diffusion H(l+1) by: H(l+1)=Ã(l) H(l), wherein H(l) is input dependency tree graph embedding of the l-th layer.

5. The system of claim 4, wherein the l-th GDT layer is configured to approximate the graph attention diffusion H(l+1) by: Z(k+1)=(1−α)A(l)Zk+αH(l);Z(0)=H(l); andH(l+1)=Zk,wherein k is an integer in a range of [0-K].

6. The system of claim 5, wherein K is a positive integer in a range of 2-12, and l is a positive integer in a range of 2-24.

7. The system of claim 6, wherein the l-th GDT layer is configured to obtain the updated embedding of the independency tree graph by: concatenating the graph attention diffusions H(l+1) of a plurality of heads to obtain concatenated attention diffusion Ĥ(l+1) by: Ĥ(l+1)=Concat(h1(l+1), . . . , hT(l+1))W0, wherein each of h1(l+1), . . . , hT(l+1) corresponds to one of the plurality of graph attention diffusions H(l+1), W0=Tdh×Tdh, T is a number of heads, dh is hidden dimensions of each head, and dh=d/T;performing {tilde over (H)}l+1=Ĥl+1+Norm(H(l)); andperforming H(l+1)=W2 (σ(W1Norm({tilde over (H)}l+1)))+{tilde over (H)}l+1, wherein W1=∈d×d and W2=∈d×d are trainable matrix, σ represents ReLU activation function, and H(l+1) is the updated embedding of the independency tree graph.

8. The system of claim 7, wherein the computer executable code comprises a classifier, and the classifier is configured to classify the aspect term by: ŷ=W2σ(W1ĤtT),wherein W2=∈C×dout and W1=∈dout×dh are learnable weight matrix, C is class number of the classification, σ is tanh activation function, Ĥt is aspect term embedding extracted from the updated embedding H(l+1), ĤtT is transpose of Ĥt, and dout is dimensions of H(l+1).

9. The system of claim 8, wherein the classifier is configured to calculate the loss function by: loss=−Σc=1Cy log ŷ+λ∥θ∥2,wherein λ is a coefficient for L2-regularization, θ are parameters that need to be regularized, and y is the label of the aspect term.

10. The system of claim 9, wherein the computer executable code is further configured to: receive a query sentence and a query aspect term from the query sentence;convert the query sentence into a query dependency tree graph;calculate a query attention matrix of the query dependency tree graph based on one-hop attention between any two of a plurality of the nodes in the query dependency tree graph;calculate a query graph attention diffusion from the query attention matrix;obtain updated query embedding of the query dependency tree graph using the query graph attention diffusion;classify the query aspect term based on the updated query embedding of the query dependency tree graph to obtain predicted classification of the query aspect term; andlabel the query aspect term with the predicted classification.

11. A method comprising: receiving, by a computing device, a sentence having an aspect term and context, the aspect term having a classification label;converting, by the computing device, the sentence into a dependency tree graph;calculating, by the computing device, an attention matrix of the dependency tree graph based on one-hop attention between any two of the plurality of the nodes in the dependency tree graph;calculating, by the computing device, graph attention diffusion from multi-hop attention between any two of the plurality of nodes in the dependency tree graph based on the attention matrix;obtaining, by the computing device, updated embedding of the dependency tree graph using the graph attention diffusion;classifying, by the computing device, the aspect term based on the updated embedding of the dependency tree graph to obtain predicted classification of the aspect term;calculating, by the computing device, loss function based on the predicted classification of the aspect term and the classification label of the aspect term; andadjusting, by the computing device, parameters of models in the computer executable code.

12. The method of claim 11, wherein the step of converting the sentence into the dependency tree graph comprises: embedding the aspect term to obtain an aspect term embedding Xt;converting the aspect term embedding to aspect term feature Hα by: α=softmax (σ(W2σ(W1XtT))) (1) and Hα=αXt (2), wherein α∈1×m, Hα ∈1×d, m is a positive integer indicating a number of embedding tokens of the aspect term, d is a positive integer indicating dimensions of embedding, XtT is the transposition of Xt, and σ denotes tanh activation function;embedding the context to obtain context embeddings Xc, each of the context embeddings corresponding to a word or a punctuation in the sentence that are not the aspect term;parsing the sentence into a dependency tree; andcombining the aspect term feature Hα, the context embeddings Xc, and the dependency tree into the dependency tree graph, wherein each node of the dependency tree graph has a feature corresponding to one of the aspect term embedding and the context embeddings.

13. The method of claim 12, wherein the step of calculating the graph attention matrix comprises: calculating an attention score si,j(l)=σ2 (v*σ1 (Whhi(l)∥Wthj(l))) for nodes i and node j in the dependency tree graph, wherein Wh, Wt ∈d×d and v∈2×d are learnable weights, hi(l) is a feature of node i at l-th graph diffusion attention (GDT) layer of a plurality of GDT layers, ∥ is a concatenation operation, σ1 is a ReLU activation function, and σ2 is a LeakyReLU activation function;obtaining attention score matrix S(l) by:

14. The method of claim 13, wherein the H(l+1) is approximated by: Z(k+1)=(1−α)A(l)Zk+αH(l);Z(0)=H(l); andH(l+1)=Zk,wherein k is an integer in a range of [0-K].

15. The method of claim 14, wherein the step of obtaining the updated embedding of the independency tree graph comprises: concatenating the graph attention diffusions H(l+1) of a plurality of heads to obtain concatenated attention diffusion Ĥ(l+1) by: Ĥ(l+1)=Concat(h1(l+1), . . . , hT(l+1))W0, wherein each of h1(l+1), . . . , hT(l+1) corresponds to one of the plurality of graph attention diffusions H(l+1), W0=Tdh×Tdh, T is a number of heads, dh is hidden dimensions of each head, and dh=d/T;performing {tilde over (H)}l+1=Ĥ(l+1)+Norm(H(l)); andperforming H(l+1)=W2 (σ(W1Norm({tilde over (H)}l+1))+{tilde over (H)}l+1, wherein W1=∈d×d and W2=∈d×d are trainable matrix, a represents ReLU activation function, and H(l+1) is the updated embedding of the independency tree graph.

16. The method of claim 15, wherein the step of classifying the aspect term comprises: ŷ=W2σ(W1ĤtT),wherein W2=∈C×dout and W1=∈dout×dh are learnable weight matrix, C is class number of the classification, σ is tanh activation function, Ĥt is aspect term embedding extracted from the updated embedding H(l+1), ĤtT is transpose of Ĥt, and dout is dimensions of H(l+1).

17. The method of claim 16, wherein the loss function is calculated by: loss=−Σc=1Cy log ŷ+λ∥θ∥2,wherein λ is a coefficient for L2-regularization, θ are parameters that need to be regularized, and y is the label of the aspect term.

18. The method of claim 17, further comprising: receiving a query sentence and a query aspect term from the query sentence;converting the query sentence into a query dependency tree graph;calculating a query attention matrix of the query dependency tree graph based on one-hop attention between any two of a plurality of the nodes in the query dependency tree;calculating a query graph attention diffusion from the query attention matrix;obtaining updated query embedding of the query dependency tree graph using the query graph attention diffusion;classifying the query aspect term based on the updated query embedding of the query dependency tree graph to obtain predicted classification of the query aspect term; andlabeling the query aspect term with the predicted classification.

19. A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of an active computing device, is configured to: receive a sentence having an aspect term and context, the aspect term having a classification label;convert the sentence into a dependency tree graph;calculate an attention matrix of the dependency tree graph based on one-hop attention between any two of a plurality of nodes in the dependency tree graph;calculate graph attention diffusion from multi-hop attention between any two of the plurality of nodes in the dependency tree graph based on the attention matrix;obtain updated embedding of the dependency tree graph using the graph attention diffusion;classify the aspect term based on the updated embedding of the dependency tree graph to obtain predicted classification of the aspect term;calculate loss function based on the predicted classification of the aspect term and the classification label of the aspect term; andadjust parameters of models in the computer executable code.

20. The non-transitory computer readable medium of claim 19, wherein the computer executable code is configured to: calculate the graph attention matrix by: calculating an attention score si,j(l)=σ2 (v*σ1 (Whhi(l)∥Wthj(l))) for nodes i and node j in the dependency tree graph, wherein Wh, Wt ∈d×d and v∈2×d are learnable weights, hi(l) is a feature of node i at l-th graph diffusion transformer (GDT) layer of a plurality of GDT layers, ∥ is a concatenation operation, σ1 is a ReLU activation function, and σ2 is a LeakyReLU activation function;obtaining attention score matrix S(l) by: