MULTI-TURN DIALOGUE SYSTEM AND METHOD BASED ON RETRIEVAL

Abstract

The multi-turn dialogue system based on retrieval includes the following modules: a representation module, a matching module, an aggregation module and a prediction module; the multi-turn dialogue method based on retrieval includes the following steps: (1) by a representation module, converting each turn of dialogue into a cascade vector of the dialogue, and converting a candidate answer into a cascade vector of the candidate answer; (2) by a matching module, dynamically absorbing context information based on a global attention mechanism, and calculating a matching vector; (3) by aggregation module, obtaining a short-term dependence information sequence and a long-term dependence information sequence; (4) by a prediction module, calculating the matching score of the context and candidate answer involved in the matching; (5) selecting a candidate answer with the highest matching score as a correct answer.

Description

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the priority to a Chinese Patent Application No. 202210649202.3, filed with the China National Intellectual Property Administration on Jun. 9, 2022 and entitled “Multi-turn dialogue system and method based on retrieval”, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present application relates to a multi-turn dialogue system and method based on retrieval, belonging to the technical field of natural language processing, in particular to the technical field of dialogue robot.

BACKGROUND

It is a challenging task in the field of artificial intelligence to create a robot that can communicate naturally with humans in the open field. At present, there are two methods to build such a dialogue robot, i.e., a generative-based method and a retrieval-based method. The generative-based method directly generates a reply based on a language model trained on a large-scale dialogue data set, while the retrieval-based method selects a best matched reply from a candidate set.

The task of the dialogue robot based on retrieval is: given a candidate set consisting of the first n turns of dialogue and several candidate answers, the model is required to select a dialogue that is most suitable for the n+1-th turn of dialogue from the candidate set, which is an important and quite challenging task. A core step of this task is to calculate a matching score of n turns of dialogues (i.e. context) and candidate answers. Some early methods were to aggregate n turns of dialogues into a dense vector through a recurrent neural network, and then calculate a cosine similarity between candidate answer representation and the dense vector, so as to select a candidate answer with the highest score as the answer. In order to avoid the loss of context information, Wu et al. proposed an SMN model (see the literature: Wu Y, Wu W, Xing C, et al Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots[C]; proceedings of the Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30-August 4, Volume 1: Long Papers, F, 2017. Association for Computational Linguistics.), which matches the candidate answers with each turn of dialogue to obtain a matching vector, and then aggregates the n matching vectors in time sequence through the recurrent neural network RNN, and then calculates a score with the aggregated vector. Later, a lot of work continued this way. However, these work have neglected that the context information in the matching phase is very important.

The method of Gu et al. (see the literature: Gu J, Li T, Liu Q, et al. Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots[C]; proceedings of the CIKM, F, 2020.) considers the problem of global information participating in interaction, but their method simply compresses the entire dialogue context and matches candidate answers to achieve the purpose of global information participating in interaction. The matching granularity of this method is too coarse, and the compression of too long dialogue into a dense vector will lose a lot of information and may introduce noise.

In summary, how to effectively use dialogue context information to improve the matching accuracy of dialogue candidate answers has become an urgent technical problem in the technical field of dialogue robot.

SUMMARY

In view of this, the present invention aims to invent a system and method based on retrieval, which can dynamically absorb the context information in the matching process of dialogue-candidate answer, and improve the matching accuracy of dialogue-candidate answer.

In order to achieve the above purpose, the present invention proposes a multi-turn dialogue system based on retrieval, which includes the following modules:

• • a representation module, configured to represent each turn of dialogue as a cascade vector E_u of a word level vector and a character level vector; represent a candidate answer r as a cascade vector E_r of a word level vector and a character level vector;
  • a matching module, formed by stacking L global interaction layers of IOI models, and configured to take the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorb context information based on a global attention mechanism, and recursively calculate a k-th layer self-attention dialogue representation Ū^k, a k-th layer self-attention candidate answer representation {circumflex over (R)}^k, a k-th layer mutual attention dialogue representation Ū^k, a k-th mutual attention candidate answer representation R^k, a k-th layer dialogue synthesis representation U^k, and a k-th layer candidate answer synthesis representation R^k, by the following formulas, to obtain a matching vector (v₁, . . . , v₁):

Û ^k =f _catt(U^k-1,U^k-1,C)

{circumflex over (R)} ^k =f _catt(R^k-1,R^k-1,C)

Ū ^k =f _catt(U^k-1,R^k-1,C)

R ^k =f _catt(R^k-1,U^k-1,C)

Ũ ^k =[U ^k-1 ,Û ^k ,Ū ^k ,U ^k-1 ⊙Ū ^k]

{tilde over (R)} ^k =[R ^k-1 ,{circumflex over (R)} ^k ,R ^k ,R ^k-1 ⊙R ^k]

U ^k=max(0,W_hŨ^k-1+b_h)

R ^k=max(0,W_h{tilde over (R)}^k-1+b_h)+R^k-1

In the formulas, U^k-1∈^m×d and R^k-1∈^n×d represent inputs of a k-th global interaction layer, wherein m and n represent the number of words contained in a current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of a first global interaction layer are U⁰=E_u, R⁰=E_r; W_h∈^4d×d and b_h are training parameters; an operator ⊙ represents a multiplication of elements; d represents the dimension of a vector;

• • C∈^lc×d represents context obtained by cascading contents of all l turns of dialogue, all l turns of dialogues contain l_c words, C can be obtained by cascading word level vectors of the l_c words;

In the formulas, f_catt( ) represents the global attention mechanism, which is specifically defined as follows:

f _catt(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)})

• • where, FNN({tilde over (Q)})=max(0,{tilde over (Q)}W_f+b_f)W_g+b_g, W_{f,g}∈^d×d and b_{f,g} are trainable parameters, Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}, wherein {circumflex over (Q)} is calculated according to the following formula:

{circumflex over (Q)}=S(Q,K,C)·K

• • where, Q∈^nq×d represents a query sequence, K∈^nk×d represents a key sequence, wherein n_q and n_k represent the number of words, S(Q,K,C)∈^nq×nk represents a similarity of Q and K in the context C; S(Q, K, C) is calculated according to the following formula:

${S\left(Q,K,C\right)}_{i,j}=\frac{\exp \left(e_{i,j}\right)}{{\sum }_{k=1}^{n_k}{e}_{i,k}}$ $e_{i,j}=W_b\tanh \left(W_c{Q}_i+W_d{K}_j+W_e\frac{C_i^q+C_j^k}{2}\right)$

• • where, W_{b,c,d,e} are trainable parameters, C_i^q represents an i-th row of C^q, and its physical meaning is fusion context information related to an i-th word in the query sequence Q; C_j^k represents a j-th row of C^k, and its physical meaning is fusion context information related to a j-th word of the key sequence K;
  • C_q∈^nq×d and C^k∈^nk×d represent context information compression vector fusing the query vector Q and context information compression vector fusing the key vector K, respectively, and are calculated according to the following formulas:

C ^q=softmax(QW₂C^T)·C

C ^k=softmax(KW₂C^T)·C

• • W₂∈^d×d are training parameters; and
  • extract a d dimension matching vector v_i from a matching image M_i of an i-th turn of dialogue by a convolutional neural network, and matching vectors from the first to the l-th turn by a convolutional neural network are represented by (v₁, . . . , v_l); the matching image M_i of the i-th turn of dialogue is calculated according to the following formula:

M _i =M _i,self |M _{i,interaction} ⊕M _i,enhanced

• • where, M_i∈^mi×n×3, ⊕ is a cascading operation, m_i is the number of words contained in the i-th turn of dialogue u_i; M_i,self, M_{i,interaction} and M_i,enhanced are calculated according to the following formulas:

$M_{i,\mathrm{self}}=\frac{{\hat{U}}_i^L·{\left({\hat{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{interaction}}=\frac{{\stackrel{\_}{U}}_i^L·{\left({\stackrel{\_}{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{enhanced}}=\frac{U_i^L·{\left(R^L\right)}^{\top }}{\sqrt{d}}$

• • an aggregation module composed of one RNN network and one Transformer network, and configured to: receive the matching vector (v₁, . . . , v_l) output by the matching module, process the matching vector by the RNN network to obtain a short-term dependence information sequence (h₁, . . . , h_l), and process the matching vector by the Transformer network to obtain a long-term dependence information sequence (g₁, . . . , g_l);
  • a prediction module: configured to calculate a matching score of the context c and the candidate answer r involved in the matching according to the short-term dependence information sequence (h₁, . . . , h_l) and long-term dependence information sequence (g₁, . . . , g_l) output by the aggregation module.

The word level vector in the representation module is obtained by a tool Word2vec; the character level vector is obtained by encoding character information through the convolutional neural network.

The specific calculation process of the short-term dependence information sequence (h₁, . . . , h_l) is:

• • obtaining l hidden layer state vectors by processing the matching vector (v₁, . . . , v_l) through a GRU model, wherein an i-th hidden layer state is:

h _i =GRU(v_i,h_i-1)

• • where, h₀ is initialized randomly.

The specific calculation process of the long-term dependence information sequence (g₁, . . . , g_l) is:

(g_l, . . . ,g_l)=MultiHead(Q,K,V)

where,

Q=V _m W ^Q , K=V _m W _K , V=V _m W ^V,

• • where, W^Q, W^K and W^V are training parameters; Multihead ( ) represents a multi-head attention function; V_m=(v₁, . . . , v_l).

The specific process of the prediction module to calculate a matching score of the context c and the candidate answer r involved in matching is calculated as follows:

• • calculating

${\hat{g}}_i=\max \left(0,W\left[\begin{array}{c}\left(g_i-h_i\right)\odot \left(g_i-h_i\right)\\ g_i\odot h_i\end{array}\right]+b\right)$

to obtain (ĝ₁, . . . , ĝ_l), wherein ⊙ represents the multiplication of elements;

• • then inputting (ĝ₁, . . . , ĝ_l) into a GRU model, to obtain:

{tilde over (g)} _i =GRU(ĝ_i,{tilde over (g)}_i-1)

• • wherein {tilde over (g)}₀ is initialized randomly; a final hidden layer state of the GRU model is {tilde over (g)}_l; calculating the matching score of the context c and the candidate answer r involved in matching based on {tilde over (g)}_l:

g(c,r)=σ({tilde over (g)}_l·w_o+b_o)

In the above formula, σ(·) represents a sigmoid function, w_o and b_o are training parameters.

The system is trained using the following loss function:

$L_{\mathrm{enhanced}}=-\sum _L^{k=1}\sum _N^{i=1}\left[y_i\log \left(g^k\left(c_i,r_i\right)\right)+\left(1-y_i\right)\log \left(1-g^k\left(c_i,r_i\right)\right)\right]$

• • where, g^k(c_l,r_l) represents a matching score calculated by using an output (U^k, R^k) of the k-th global interaction layer in the matching module for an i-th training sample (c_i, r_i, y_i); c_i represents a session of the i-th training sample, r_i represents a candidate answer of the i-th training sample, y_i represents a true matching result of the i-th training sample.

The present invention also proposes a multi-turn dialogue method based on retrieval, comprising:

• • (1) by a representation module, converting each turn of dialogue into a cascade vector E_u of the dialogue, and converting a candidate answer r into a cascade vector E_r of the candidate answer; the cascade vector E_u of the dialogue is obtained by cascading a word level vector and a character level vector in the dialogue; the cascade vector E_r of the candidate answer is obtained by cascading a word level vector and a character level vector in the candidate answer; the word level vector is obtained by a tool Word2vec; the character level vector is obtained by encoding character information through a convolutional neural network;
  • (2) by a matching module, taking the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorbing context information based on a global attention mechanism, and recursively calculating a k-th layer self-attention dialogue representation , a k-th layer self-attention candidate answer representation , a k-th layer mutual attention dialogue representation , a k-th mutual attention candidate answer representation , a k-th layer dialogue synthesis representation , and a k-th layer candidate answer synthesis representation , by the following formulas, to obtain a matching vector (v₁, . . . , v_l):

Û ^k =f _catt(U^k-1,U^k-1,C)

{circumflex over (R)} ^k =f _catt(R^k-1,R^k-1,C)

Ū ^k =f _catt(U^k-1,R^k-1,C)

R ^k =f _catt(R^k-1,U^k-1,C)

Ũ ^k =[U ^k-1 ,Û ^k ,Ū ^k ,U ^k-1 ⊙Ū ^k]

{tilde over (R)} ^k =[R ^k-1 ,{circumflex over (R)} ^k ,R ^k ,R ^k-1 ⊙R ^k]

U ^k=max(0,W_hŨ^k-1+b_h)

R ^k=max(0,W_h{tilde over (R)}^k-1+b_h)+R^k-1

• • in the formulas, U_k-1∈^m×d and R^k-1∈^n×d represent inputs of a k-th global interaction layer, wherein m and n represent the number of words contained in a current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of a first global interaction layer is U⁰=E_u, R⁰=E_r; W_h∈^4d×d and b_h are training parameters; the operator ⊙ represents a multiplication of elements; d represents the dimension of a vector;
  • C∈^lc×d represents context obtained by cascading contents of all l turns of dialogues, all l turns of dialogues contain l_c words, C can be obtained by cascading word level vectors of the l_c words;
  • in the formulas, f_catt( ) represents the global attention mechanism, which is specifically defined as follows:

f _catt(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)})

• • where FNN({tilde over (Q)})=max(0,{tilde over (Q)}W_f+b_f)W_g+b_g, W_{f,g}∈^d×d and b_{f,g} are trainable parameters, Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}, wherein {circumflex over (Q)} is calculated according to the following formula:

{circumflex over (Q)}=S(Q,K,C)·K

• • where, Q∈^nq×d represents a query sequence, K∈^nk×d represents a key sequence, wherein n_q and n_k represent the number of words, S(Q,K,C)∈^nq×nk represents a similarity of Q and K in the context C; S(Q, K, C) is calculated according to the following formula:

${S\left(Q,K,C\right)}_{i,j}=\frac{\exp \left(e_{i,j}\right)}{{\sum }_{k=1}^{n_k}{e}_{i,k}}$ $e_{i,j}=W_b\tanh \left(W_c{Q}_i+W_d{K}_j+W_e\frac{C_i^q+C_j^k}{2}\right)$

• • where, W_{b,c,d,e} are trainable parameters, C_i^q represents an i-th row of C^q, and its physical meaning is the fusion context information related to the i-th word in the query sequence Q; C_j^k represents a j-th row of C^k, and its physical meaning is fusion context information related to a j-th word of the key sequence K;
  • C_q∈^nq×d and C_k∈^nk×d represent context information compression vector fusing the query vector Q and context information compression vector fusing the key vector K, respectively, and they are calculated according to the following formula:

C ^q=softmax(QW_aC^T)·C

C ^k=softmax(KW_aC^T)·C

• • W_a∈^d×d are training parameters; and
  • extracting a d dimension matching vector v_i from a matching image M_i of the i-th turn of dialogue by a convolutional neural network, and matching vectors from the first to the l-th turn of dialogues are represented by (v₁, . . . , v_l); the matching image M_i of the i-th turn of dialogue is calculated according to the following formula:

M _i =M _i,self ⊕M _{i,interaction} ⊕M _i,enhanced

• • where, M_i∈^mi×n×3, ⊕ is a cascading operation, m_i is the number of words contained in the i-th turn of dialogue u_i; M_i,self, M_{i,interaction} and M_i,enhanced are calculated according to the following formulas:

$M_{i,\mathrm{self}}=\frac{{\hat{U}}_i^L·{\left({\hat{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{interaction}}=\frac{{\stackrel{\_}{U}}_i^L·{\left({\stackrel{\_}{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{enhanced}}=\frac{U_i^L·{\left(R^L\right)}^{\top }}{\sqrt{d}}.$

• • (3) receiving, by an aggregation module, the matching vector (v₁, . . . , v_l) output by the matching module, processing the matching vector by the RNN network of the aggregation module to obtain a short-term dependence information sequence (h₁, . . . , h_l), and processing the matching vector by the Transformer network of the aggregation module to obtain a long-term dependence information sequence (g₁, . . . , g_l);
  • wherein a specific calculation process of the short-term dependence information sequence (h₁, . . . , h_l) is:
  • obtaining/hidden layer state vectors by processing the matching vector (v₁, . . . , v_l) through a GRU model, wherein an i-th hidden layer state is:

h _i =GRU(v_i,h_i-1)

• • where, h₀ is initialized randomly.
  • a specific calculation process of the long-term dependence information sequence (g₁, . . . , g_l) is:

(g₁, . . . ,g_l)=MultiHead(Q,K,V)

• • where,

Q=V _m W ^Q , K=V _m W ^K , V=V _m W ^V,

• • where W^Q, W^K and W^V are training parameters; u represents a multi-head attention function; V_m=(V₁, . . . , V_l).
  • (4) calculating, by a prediction module, a matching score of the context c and the candidate answer involved in matching according to the short-term dependence information sequence (h₁, . . . , h_l) and the long-term dependence information sequence (g₁, . . . , g_l) output by the aggregation module, wherein the calculating includes:
  • calculating

${\hat{g}}_i=\max \left(0,W\left[\begin{array}{c}\left(g_i-h_i\right)\odot \left(g_i-h_i\right)\\ g_i\odot h_i\end{array}\right]+b\right)$

to obtain (ĝ₁, . . . , ĝ_l), wherein ⊕ represents the multiplication of elements;

• • then inputting (ĝ₁, . . . , ĝ_l) into a GRU model, to obtain:

{tilde over (g)} _i =GRU({tilde over (g)}_i,{tilde over (g)}_i-1)

• • wherein {tilde over (g)}_σ is initialized randomly; a final hidden layer state of the GRU model is {tilde over (g)}_l;
  • calculating the matching score of the context c and the candidate answer r involved in matching based on {tilde over (g)}_l:

g(c,r)=σ({tilde over (g)}_l·w_o+b_o)

• • where, σ(·) represents a sigmoid function, w_o and b_o are training parameters.
  • (5) selecting a candidate answer with a highest matching score as a correct answer.

The beneficial effect of the present invention is that the system and method of the present invention extend a general attention mechanism to a global attention mechanism, and dynamically absorb the context information in the dialogue-candidate answer matching process. The system of the present invention can simultaneously capture the short-term dependence and long-term dependence of the matching information sequence, and effectively improve the matching accuracy of dialogue-candidate answer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a module diagram of a multi-turn dialogue retrieval-based system proposed by the present invention.

FIG. 2 is a flow diagram of a multi-turn dialogue retrieval-based method proposed by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in combination with the drawings.

Referring to FIG. 1, a multi-turn dialogue system based on retrieval proposed by the present invention is introduced. The system includes the following modules:

• • a representation module, configured to represent each turn of dialogue as a cascade vector E_u=[e_u,1, . . . , e_u,m] of a word level vector and a character level vector; to represent a candidate answer r as a cascade vector [e_r,1, . . . , e_r,u] of a word level vector and a character level vector;
  • a matching module, formed by stacking L (in this embodiment, L takes the value of 4) global interaction layers of IOI models (For details of IOI model, see Tao C, Wu W, Xu C, et al. One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues[C]; proceedings of the ACL, F, 2019.), and configured to take the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorb context information based on a global attention mechanism, and recursively calculate a k-th layer self-attention dialogue representation Ū^k, a k-th layer self-attention candidate answer representation {circumflex over (R)}^k, a k-th layer mutual attention dialogue representation Ū^k, a k-th mutual attention candidate answer representation R^k, a k-th layer dialogue synthesis representation U^k, and a k-th layer candidate answer synthesis representation R^k, to obtain a matching vector (v₁, . . . , v_l).

The attention mechanism is the basis of the global interaction layer. For details of attention mechanism, see Vaswani A, Shazeer N, Parmar N, et al. Attention is All you Need[C]; proceedings of the NIPS, F, 2017.

The specific calculation process is as follows:

Û ^k =f _catt(U^k-1,U^k-1,C)

{circumflex over (R)} ^k =f _catt(R^k-1,R^k-1,C)

Ū ^k =f _catt(U^k-1,R^k-1,C)

R ^k =f _catt(R^k-1,U^k-1,C)

Ũ ^k =[U ^k-1 ,Û ^k ,Ū ^k ,U ^k-1 ⊙Ū ^k]

{tilde over (R)} ^k =[R ^k-1 ,{circumflex over (R)} ^k ,R ^k ,R ^k-1 ⊙R ^k]

U ^k=max(0,W_hŨ^k-1+b_h)

R ^k=max(0,W_h{tilde over (R)}^k-1+b_h)+R^k-1

In the above formulas, U^k-1∈^m×d and R^k-1∈^n×d represent inputs of a k-th global interaction layer, wherein m and n represent the number of words contained in a current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of the first global interaction layer is U⁰=E_u, R⁰=E_r; W_h∈^4d×d and b_h are training parameters; the operator ⊙ represents a multiplication of elements; d represents the dimension of a vector

• • C∈^lc×d represents context obtained by cascading contents of all l turn of dialogue. All l turns of dialogues contain l_c words, C can be obtained by cascading word level vectors of the l_c words;

In the above formula, f_catt( ) represents the described global attention mechanism, which is specifically defined as follows:

f _catt(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)})

In the above formula, FNN({tilde over (Q)})=max(0,{tilde over (Q)}W_f+b_f)W_g+b_g, wherein W_{f,g}∈^d×d and b_{f,g} are trainable parameters, Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}.

In this embodiment, Q and {circumflex over (Q)} are mixed using a residual connection used by He et al. (For details, see He K, Zhang X, Ren S, et al. Deep Residual Learning for Image Recognition[C]; proceedings of the CVPR 2016, F, 2016.) in the present invention to obtain a new Q. In order to prevent gradient explosion or gradient disappearance, the present invention uses a layer normalization (For detail, see: Ba L J, Kiros J R, Hinton G E. Layer Normalization[J]. CoRR, 2016.).

Wherein {circumflex over (Q)} is calculated according to the following formula:

{circumflex over (Q)}=S(Q,K,C)·K

In the above formula, Q∈^nq×d represents a query sequence, K∈^nk×d represents a key sequence, wherein n_q and n_k represent the number of words, S(Q,K,C)∈^nq×nk represents the similarity of Q and K in the context C; S(Q, K, C) is calculated according to the following formula:

${S\left(Q,K,C\right)}_{i,j}=\frac{\exp \left(e_{i,j}\right)}{{\sum }_{k=1}^{n_k}{e}_{i,k}}$ $e_{i,j}=W_b\tanh \left(W_c{Q}_i+W_d{K}_j+W_e\frac{C_i^1+C_j^k}{2}\right)$

In the above formula, W_{b,c,d,e} are trainable parameters, C_i^q represents the i-th row of C^q, and its physical meaning is the fusion context information related to the i-th word in the query sequence Q; C_l^k represents the j-th row of C^k, and its physical meaning is the fusion context information related to the j-th word of the key sequence K;

• • C^q∈^nq×d and C^k∈^nk×d represent context information compression vector fusing the query vector Q and context information compression vector fusing the key vector K, respectively, and are calculated according to the following formula:

C ^q=softmax(QW_aC^T)·C

C ^k=softmax(KW_aC^T)·C

• • W_a∈^d×d are training parameters; softmax ( ) represents a softmax function.

A convolutional neural network is used to extract a d dimension matching vector v_i from a matching image M_i of the i-th turn of dialogue, and the matching vector from the first to l-th turn of dialogues are represented by (v₁, . . . , v_l); the matching image M_i of the i-th turn of dialogue is calculated according to the following formula:

M _i =M _i,self ⊕M _{i,interaction} ⊕M _i,enhanced

In the above formula, M_i∈^mi×n×3, ⊕ is a cascading operation, m_i is the number of words contained in the i-th turn of dialogue u_i; M_i,self, M_{i,interaction} and M_i,enhanced are calculated according to the following formulas:

$M_{i,\mathrm{self}}=\frac{{\hat{U}}_i^L·{\left({\hat{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{interaction}}=\frac{{\stackrel{\_}{U}}_i^L·{\left({\stackrel{\_}{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{enhanced}}=\frac{U_i^L·{\left(R^L\right)}^{\top }}{\sqrt{d}}.$

The aggregation module: this module is composed of one RNN network and one Transformer network, and configured to: receive the matching vector (v₁, . . . , v_l) output by the matching module, process the matching vector by the RNN network to obtain a short-term dependence information sequence (h₁, . . . , h_l), and process the matching vector by the Transformer network to obtain a long-term dependence information sequence (g₁, . . . , g_l).

In this embodiment, an encoder in Transformer (For detail, see Vaswani A, Shazeer N, Parmar N, et al. Attention is All you Need[C]; proceedings of the NIPS, F, 2017.) captures the long-term dependence information in the matching vector (v₁, . . . , v_l).

The prediction module is configured to calculate a matching score of the context c and the candidate answer r involved in matching according to the short-term dependence information sequence (h₁, . . . , h_l) and the long-term dependence information sequence (g₁, . . . , g_l) output by the aggregation module.

The word level vector described in the representation module is obtained by the tool Word2vec (see Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and their Compositionality[C]; proceedings of the NIPS 2013, F, 2013.); the character level vector is obtained by encoding character information through the convolutional neural network. For the convolutional neural network used in the embodiment, see Lee K, He L, Lewis M, et al. End-to-end Neural Coreference Resolution [Z]. EMNLP. 2017.

The specific calculation process of the short-term dependence information sequence (h₁, . . . , h_l) is:

• • obtaining l hidden layer state vectors by processing the matching vector (v₁, . . . , v_l) through a GRU mode, wherein the i-th hidden layer state is:

h _l =GRU(v_l,h_1-l)

• • where, h₀ is initialized randomly.

The specific calculation process of the long-term dependence information sequence (g₁, . . . , g_l) is:

(g₁, . . . ,g_l)=MultiHead(Q,K,V)

where,

Q=V _m W ^Q ,K=V _m W ^K ,V=V _m W ^V,

• • wherein W^Q, W^K and W^V are training parameters; Multihead( ) represents a multi-head attention function; V_m=(V₁, . . . , V_l).

The specific process of the prediction module to calculate the matching score of the context c and the candidate answer r involved in matching is calculated as follows:

• • calculating

${\hat{g}}_i=\max \left(0,W\left[\begin{array}{c}\left(g_i-h_i\right)\odot \left(g_i-h_i\right)\\ g_i\odot h_i\end{array}\right]+b\right)$

to obtain (ĝ₁, . . . ,ĝ_l), wherein ⊕ represents the multiplication of elements;

• • then inputting (ĝ₁, . . . , ĝ_l) into a GRU model, to obtain:

{tilde over (g)} _l =GRU(ĝ_i,{tilde over (g)}_i-1)

• • wherein {tilde over (g)}_o is initialized randomly; the final hidden layer state of the GRU model is {tilde over (g)}₁
  • calculating the matching score of the context c and the candidate answer r involved in matching based on {tilde over (g)}_l:

g(c,r)=σ({tilde over (g)}_l·w_o+b_o)

In the above formula, σ(·) represents a sigmoid function, w_o and b_o are training parameters.

The system is trained using the following loss function:

$L_{\mathrm{enhanced}}=-\sum _L^{k=1}\sum _N^{i=1}\left[y_i\log \left(g^k\left(c_i,r_i\right)\right)+\left(1-y_i\right)\log \left(1-g^k\left(c_i,r_i\right)\right)\right]$

• • where, g^k(c_i,r_i) represents a matching score calculated by using the output (U^k,R^k) of the k-th global interaction layer in the matching module for the i-th training sample (c_i, r_i, y_i); c_i represents a session of the i-th training sample, r_i represents a candidate answer of the i-th training sample, y_i represents a true matching result of the i-th training sample.

Referring to FIG. 2, the present invention also proposes a multi-turn dialogue method based on retrieval, comprising:

• • (1) by a representation module, converting each turn of dialogue into a cascade vector E_u of the dialogue, and converting a candidate answer r into a cascade vector E_r of the candidate answer; the cascade vector E_u of the dialogue is obtained by cascading a word level vector and a character level vector in the dialogue; the cascade vector E_r of the candidate answer is obtained by cascading a word level vector and a character level vector in the candidate answer; the word level vector is obtained by a tool Word2vec; the character level vector is obtained by encoding character information through a convolutional neural network;
  • (2) by a matching module, taking the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorbing context information based on a global attention mechanism, and recursively calculating a k-th layer self-attention dialogue representation , a k-th layer self-attention candidate answer representation , a k-th layer mutual attention dialogue representation , a k-th mutual attention candidate answer representation , a k-th layer dialogue synthesis representation , and a k-th layer candidate answer synthesis representation , by the following formulas, to obtain a matching vector (v₁, . . . , v_l):

Û ^k =f _catt(U^k-1,U^k-1,C)

{circumflex over (R)} ^k =f _catt(R^k-1,R^k-1,C)

Ū ^k =f _catt(U^k-1,R^k-1,C)

R ^k =f _catt(R^k-1,U^k-1,C)

Ũ ^k =[U ^k-1 ,Û ^k ,Ū ^k ,U ^k-1 ⊙Ū ^k]

{tilde over (R)} ^k =[R ^k-1 ,{circumflex over (R)} ^k ,R ^k ,R ^k-1 ⊙R ^k]

U ^k=max(0,W_hŨ^k-1+b_h)

R ^k=max(0,W_h{tilde over (R)}^k-1+b_h)+R^k-1

In the above formula, U^k-1∈^m×d and R^k-1∈^n×d represent inputs of the k-th global interaction layer, wherein m and n represent the number of words contained in the current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of the first global interaction layer are U⁰=E_u, R⁰=E^r; W_h∈^4d×d and b_h are training parameters; the operator ⊕ represents a multiplication of elements; d represents the dimension of a vector

• • C∈^lc×d represents context obtained by cascading the contents of all l turns of dialogues. All l turns of dialogues contain l_c words, C can be obtained by cascading word level vectors of the l_c words;

In the above formula, f_catt( ) represents the described global attention mechanism, which is specifically defined as follows:

f _catt(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)})

In the above formula, FNN({tilde over (Q)})=max(0,{tilde over (Q)}W_f+b_f)W_g+b_g, wherein W_{f,g}∈^d×d and b_{f,g} are trainable parameters, Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}, wherein {circumflex over (Q)} is calculated according to the following formula:

{circumflex over (Q)}=S(Q,K,C)·K

In the above formula, Q∈^nq×d represents a query sequence, K∈^nk×d represents a key sequence, wherein n_q and n_k represent the number of words, S(Q,K,C)∈^nq×nk represents the similarity of Q and K in the context C; S(Q, K, C) is calculated according to the following formula:

${S\left(Q,K,C\right)}_{i,j}=\frac{\exp \left(e_{i,j}\right)}{{\sum }_{k=1}^{n_k}{e}_{i,k}}$ $e_{i,j}=W_b\tanh \left(W_c{Q}_i+W_d{K}_j+W_e\frac{C_i^1+C_j^k}{2}\right)$

In the above formula, W_{b,c,d,e} are trainable parameters, C_i^q represents the i-th row of C^q, and its physical meaning is the fusion context information related to the i-th word in the query sequence Q; C_j^k represents the j-th row of C^k, and its physical meaning is the fusion context information related to the j-th word of a key sequence K;

• • C^q∈^nq×d and C^k∈^nk×d represent context information compression vector fusing the query vector Q and context information compression vector fusing the key vector K, respectively, and they are calculated according to the following formulas:

C ^q=softmax(QW_aC^T)·C

C ^k=softmax(KW_aC^T)·C

• • W_a∈^d×d are training parameters; and
  • extracting a d dimension matching vector v_i from a matching image M_i of the i-th turn of dialogue by a convolutional neural network, and the matching vectors from the first to the l-th turn of dialogues are represented by (v₁, . . . , v_l); the matching image M_i of the i-th turn of dialogue is calculated according to the following formula:

M _i =M _i,self ⊕M _{i,interaction} ⊕M _i,enhanced

In the above formula, M_i∈^mi×n×3, ⊕ is a cascading operation, m_i is the number of words contained in the i-th turn of dialogue u_i; M_i,self, M_{i,interaction} and M_i,enhanced are calculated according to the following formulas:

$M_{i,\mathrm{self}}=\frac{{\hat{U}}_i^L·{\left({\hat{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{interaction}}=\frac{{\stackrel{\_}{U}}_i^L·{\left({\stackrel{\_}{R}}^L\right)}^{\top }}{\sqrt{d}}$ $M_{i,\mathrm{enhanced}}=\frac{U_i^L·{\left(R^L\right)}^{\top }}{\sqrt{d}}.$

• • (3) receiving, by an aggregation module, the matching vector (v₁, . . . , v_l) output by the matching module, processing the matching vector by the RNN network of the aggregation module to obtain a short-term dependence information sequence (h₁, . . . , h_l), and processing the matching vector by the Transformer network of the aggregation module to obtain a long-term dependence information sequence (g₁, . . . , g_l);

The specific calculation process of the short-term dependence information sequence (h₁, . . . , h_l) is:

• • obtaining l hidden layer state vectors by processing the matching vector (v₁, . . . , v_l) through a GRU model, wherein the i-th hidden layer state is:

h _l =GRU(v_l,h_l-1)

• • where, h₀ is initialized randomly.

The specific calculation process of the long-term dependence information sequence (g₁, . . . , g_l) is:

(g₁, . . . , g_l)=MultiHead(Q,K,V)

In the above formula,

Q=V _m W ^Q , K=V _m W _K , V=V _m W ^V,

wherein W^Q, W^K and W^V are training parameters; Multihead ( ) represents a multi-head attention function; V_m=(v₁, . . . , v_l).

• • (4) calculating, by a prediction module, a matching score of the context c and the candidate answer involved in matching according to the short-term dependence information sequence (h₁, . . . , h_l) and the long-term dependence information sequence (g₁, . . . , g_l) output by the aggregation module, wherein the calculating includes:
  • calculating

${\hat{g}}_i=\max \left(0,W\left[\begin{array}{c}\left(g_i-h_i\right)\odot \left(g_i-h_i\right)\\ g_i\odot h_i\end{array}\right]+b\right)$

to obtain (ĝ₁, . . . , {tilde over (g)}_l), wherein ⊕ represents the multiplication of elements;

• • then inputting (ĝ₁, . . . , ĝ_l) into a GRU model, to obtain:

{tilde over (g)} _i =GRU(ĝ_i,{tilde over (g)}_i-1)

• • wherein {tilde over (g)}₀ is initialized randomly; a final hidden layer state of the GRU model is {tilde over (g)}₁;
  • calculating the matching score of the context c and the candidate answer r involved in matching based on {tilde over (g)}_l:

g(c,r)=σ({tilde over (g)}_l·w_o+b_o)

In the above formula, σ(·) represents a sigmoid function, w_o and b_o are training parameters.

• • (5) Selecting a candidate answer with the highest matching score as a correct answer.

The inventor has conducted a lot of experiments on the proposed system and method on three widely used multi-turn dialogue retrieval data sets. The three data sets are Ubuntu dialogue data sets (see Lowe R, Pow N, Serban I, et al. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems[C]; proceedings of the SIGDIAL 2015, F, 2015.), Douban Dialog Data sets (see Wu Y, Wu W, Xing C, et al. Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots[C]; proceedings of the Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30-August 4, Volume 1: Long Papers, F, 2017. Association for Computational Linguistics.), and E-commerce dialogue data sets (see Zhang Z, Li J, Zhu P, et al. Modeling Multi-turn Conversation with Deep Utterance Aggregation[C]; proceedings of the COLING, F, 2018.). The statistical information of these data sets are shown in Table 1.

TABLE 1
statistical information of three multiple turn dialogue data sets
	Ubuntu	Douban	E-commerce
Data set	Train	Valid	Test	Train	Valid	Test	Train	Valid	Test
Number of dialogue-candidate	1M	500K	500K	1M	50K	50K	1M	10K	10K
answer pairs
Number of candidate answers	2	10	10	2	2	10	2	2	10
Average number of turns of dialogues	10.13	10.11	10.11	6.69	6.75	6.45	5.51	5.48	5.64
Average number of words per	11.35	11.34	11.37	18.56	18.50	20.74	7.02	6.99	7.11
turn of dialogue

In the experiment, the word level vector of English text is a 200 dimensional word vector obtained by word2vec, and the character level vector of English text is a 100 dimensional vector; The experimental results show that the method proposed by the present invention is effective and feasible, and the indicators are obviously superior to other methods.

Claims

1. A multi-turn dialogue system based on retrieval, comprising the following modules: a representation module, configured to represent each turn of dialogue as a cascade vector Eu of a word level vector and a character level vector, and represent a candidate answer r as a cascade vector Er of a word level vector and a character level vector;a matching module, formed by stacking L global interaction layers of IOI models, and configured to take the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorb context information based on a global attention mechanism, and recursively calculate a k-th layer self-attention dialogue representation Ūk, a k-th layer self-attention candidate answer representation {circumflex over (R)}k, a k-th layer mutual attention dialogue representation Ūk, a k-th mutual attention candidate answer representation Rk, a k-th layer dialogue synthesis representation Uk, and a k-th layer candidate answer synthesis representation Rk, by the following formulas, to obtain a matching vector (v1, . . . , v1): Ûk=fcatt(Uk-1,Uk-1,C){circumflex over (R)}k=fcatt(Rk-1,Rk-1,C)Ūk=fcatt(Uk-1,Rk-1,C)Rk=fcatt(Rk-1,Uk-1,C)Ũk=[Uk-1,Ûk,Ūk,Uk-1⊙Ūk]{tilde over (R)}k=[Rk-1,{circumflex over (R)}k,Rk,Rk-1⊙Rk]Uk=max(0,WhŨk-1+bh)Rk=max(0,Wh{tilde over (R)}k-1+bh)+Rk-1 in the formulas, Uk-1∈m×d and Rk-1∈n×d represent inputs of a k-th global interaction layer, wherein m and n represent the number of words contained in a current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of a first global interaction layer are U0=Eu, R0=Er; Wh∈4d×d and bh are training parameters; an operator ⊙ represents a multiplication of elements; d represents a dimension of a vector;C∈lc×d represents context obtained by cascading contents of all l turns of dialogues; all l turns of dialogues contain lc words, and C can be obtained by cascading word level vectors of the lc words;in the formulas, fcatt( ) represents the global attention mechanism, which is specifically defined as follows: fcatt(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)}),where, FNN({tilde over (Q)})=max(0,{tilde over (Q)}Wf+bf)Wgbg; W{f,g}∈d×d and b{f,g} are trainable parameters; Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}, wherein {circumflex over (Q)} is calculated according to the following formula: {circumflex over (Q)}=S(Q,K,C)·K where, Q∈nq×d represents a query sequence, K∈nk×d represents a key sequence, wherein nq and nk represent the number of words, S(Q,K,C)∈nq×nk represents a similarity of Q and K in the context C; S(Q, K, C) is calculated according to the following formulas:

2. The multi-turn dialogue system based on retrieval of claim 1, wherein the word level vector in the representation module is obtained by a tool Word2vec; the character level vector is obtained by encoding character information through the convolutional neural network.

3. The multi-turn dialogue system based on retrieval of claim 1, wherein the system is trained using the following loss function:

4. A multi-turn dialogue method based on retrieval, comprising: (1) by a representation module, converting each turn of dialogue into a cascade vector Eu of the dialogue, and converting a candidate answer r into a cascade vector Er of the candidate answer; the cascade vector Eu of the dialogue is obtained by cascading a word level vector and a character level vector in the dialogue; the cascade vector Er of the candidate answer is obtained by cascading a word level vector and a character level vector in the candidate answer; the word level vector is obtained by a tool Word2vec; the character level vector is obtained by encoding character information through a convolutional neural network;(2) by a matching module, taking the cascade vector of the dialogue and the cascade vector of the candidate answer output by the representation module as an input, dynamically absorbing context information based on a global attention mechanism, and recursively calculating a k-th layer self-attention dialogue representation Ûk, a k-th layer self-attention candidate answer representation {circumflex over (R)}k, a k-th layer mutual attention dialogue representation Ūk, a k-th mutual attention candidate answer representation Rk, a k-th layer dialogue synthesis representation Uk, and a k-th layer candidate answer synthesis representation Rk, by the following formulas, to obtain a matching vector (v1, . . . , vl): Ûk=fcatt(Uk-1,Uk-1,C){circumflex over (R)}k=fcatt(Rk-1,Rk-1,C)Ūk=fcatt(Uk-1,Rk-1,C)Rk=fcatt(Rk-1,Uk-1,C)Ũk=[Uk-1,Ûk,Ūk,Uk-1⊙Ūk]{tilde over (R)}k=[Rk-1,{circumflex over (R)}k,Rk,Rk-1⊙Rk]Uk=max(0,WhŨk-1+bh)Rk=max(0,Wh{tilde over (R)}k-1+bh)+Rk-1 in the formulas, Uk-1∈m×d and Rk-1∈n×d represent inputs of a k-th global interaction layer, wherein m and n represent the number of words contained in a current turn of dialogue and the number of words contained in the candidate answer, respectively, and inputs of a first global interaction layer are U0=Eu, R0=Er; Wh∈4×d and bh are training parameters; an operator ⊕ represents a multiplication of elements; d represents a dimension of a vector;C∈lc×d represents context obtained by cascading contents of all l turns of dialogues; all l turns of dialogues contain lc words, C can be obtained by cascading word level vectors of the lc words;in the formulas, fcatt( ) represents the global attention mechanism, which is specifically defined as follows: fcatt>(Q,K,C)={tilde over (Q)}+FNN({tilde over (Q)})where, FNN({tilde over (Q)})=max(0,{tilde over (Q)}Wf+bfWg+bg, W{f,g}∈d×d and b{f,g} are trainable parameters, Q and {circumflex over (Q)} are mixed using a residual connection to obtain {tilde over (Q)}, wherein {circumflex over (Q)} is calculated according to the following formula: {circumflex over (Q)}=S(Q,K,C)·K where, Q∈nq×d represents a query sequence, K∈nk×d represents a key sequence, wherein nq and nk represent the number of words, S(Q,K,C)∈nq×nk represents a similarity of Q and K in the context C; S(Q, K, C) is calculated according to the following formula: