SEQUENCE RECOMMENDATION METHOD BASED ON EXTRACTING AND MODELING OF COMPLEX MULTI-MODE USER INTERESTS

Abstract

A sequence recommendation method based on extraction and modeling of complex multi-mode user interests is provided, including: obtaining long-term and short-term embedding sequences; obtaining updated long-term and short-term embedding sequences through the long-term and short-term embedding sequences; with embedding vectors of last items in the updated long-term and short-term embedding sequences as long-term and short-term dynamic interests of a user, obtaining long-term and short-term static interests of the user through weighted calculation; concatenating the long-term and short-term dynamic interests and the long-term and short-term static interests, and performing nonlinear change to obtain long-term and short-term evolutionary interests of the user; obtaining a dynamic interest, a static interest and an evolutionary interest of the user through element-wise summation; performing weighted summation to obtain a fused user interest; calculating a product of the fused interest with embedding of each item as a recommendation score of each item.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims the benefit and priority of Chinese Patent Application No. 202211505338.3 filed with the China National Intellectual Property Administration on Nov. 29, 2022, the disclosure of which is incorporated by reference herein in its entirety as part of the present application.

TECHNICAL FIELD

The present disclosure belongs to the field of data mining and recommendation systems, and in particular, relates to a sequence recommendation method based on extracting and modeling of complex multi-mode user interests.

BACKGROUND

With the rapid development of mobile computing technology, the contact between people and devices becomes easier. In the process of digitalization, massive services and data are produced, and users will inevitably face the dilemma of finding the required content from massive data, which is referred to as information overload. A personalized recommendation system solves the problem of information overload by modeling user interests and recommending related content. In particular, the personalized recommendation system can help users find products/contents/projects they are interested in from massive data and create opportunities for product providers to increase their income.

Usually, users access online items in a certain order. Therefore, sequence recommendation has become a hot topic in the construction of the recommendation system in academic circles and industry circles. Given a historical item interaction sequence of a user, sequence recommendation aims to predict a next item that the user may be interested in.

Sequence recommendation takes the sequential item interaction sequence as an input. At present, the methods of sequence recommendation by domestic and international researchers can be mainly divided into three categories, namely, matrix decomposition-based method, Markov chain-based method and deep learning-based method. The matrix decomposition-based method relies on time sequence matrix decomposition to mine dynamic interests of the user. The Markov chain-based method uses first-order or higher-order Markov chains to learn changes of long-term interests and short-term interests of the user. Inspired by the advantages of a natural language processing method in sequence modeling, the deep learning-based method is used to enhance feature learning. The method based on Convolutional Neural Network (CNN), such as Caser, uses CNN to learn an item embedding sequence. The method based on Recurrent Neural Network (RNN) uses RNN or variants of RNN such as a Gated Recurrent Unit (GRU) and a Long Short-Term Memory Network (LSTM) for sequence recommendation. Recently, because Graph Neural Network (GNN) can effectively learn a high-order relationship between items, researchers use GNN for sequence recommendation tasks. SR-GNN learns item embedding by applying GNN to graphs constructed based on item sequences. SURGE uses GNN to dynamically extract user interests from noisy sequences. In addition, the attention-based method such as SASRec uses a self-attention mechanism to adaptively select related items to model user interests. TLSAN learns long-term and short-term interests through an attention network. Generally speaking, the deep learning-based method is superior to the other two kinds of methods.

The existing sequence recommendation methods are usually divided into long-term interests and short-term interests when modeling user interests. The main difference between the long-term interest and the short-term interest lies in the different sequence lengths used for interest mining. However, with the change of the length, the user interests will also change, so that the existing methods based on long-term and short-term interests cannot accurately model the representation of the user interests.

SUMMARY

In view of the defects of the existing sequence recommendation for modeling user interests from the perspectives of long-term interests and short-term interests, the present disclosure proposes a sequence recommendation method based on extraction and modeling of complex multi-mode user interests from the perspectives of dynamic interests and static interests, and considers evolutionary interests in the dynamic and static interest modeling process to enhance feature modeling, thereby realizing more accurate personalized sequence recommendation of the user.

The present disclosure provides a sequence recommendation method based on extraction and modeling of complex multi-mode user interests, which includes the following specific steps:

Step 1, acquiring a historical item interaction sequence of a user, and selecting a latest sequence with a length of m as a long-term sequence and a latest sequence with a length of n as a short-term sequence, where m>n is required; based on a self-learning item embedding matrix F∈^k×d, items involved in the sequences are embedded to obtain a long-term embedding sequence F_l and a short-term embedding sequence F_s;

Step 2, inputting the long-term embedding sequence F_l and the short-term embedding sequence F_s into two independent multi-head self-attention modules, respectively, to obtain an updated long-term embedding sequence Ê_l and an updated short-term embedding sequence Ê_s;

Step 3, with an embedding vector ê_l^m of a last item in the updated long-term embedding sequence as a long-term dynamic interest p_l^d of the user, calculating attention weights of the updated long-term embedding sequence Ê_l to the embedding vector of the last item, and performing weighted summation to obtain a long-term static interest p_l^x of the user, and similarly, obtaining a short-term dynamic interest p_s^d and a short-term static interest p_s^x based on the updated short-term embedding sequence;

Step 4, concatenating the long-term dynamic interest p_l^d and the long-term static interest p_l^x, and performing nonlinear change to obtain a long-term evolutionary interest p_l^y of the user, and similarly, obtaining a short-term evolutionary interest p_s^y of the user based on the short-term dynamic interest p_s^d and the short-term static interest p_s^x;

Step 5, obtaining a dynamic interest p^d of the user through element-wise summation of the long-term dynamic interest p_l^d and the short-term dynamic interest p_s^d; and similarly, obtaining a static interest p^x and an evolutionary interest p^y of the user;

Step 6, calculating attention weights of the dynamic interest p^d, the static interest p^x and the evolutionary interest p^y to the embedding vector of the last item, and performing weighted summation to obtain a fused user interest p;

Step 7, calculating a product of p with embedding F of each item as a recommendation score of each item, and recommending top items with highest scores for the user.

The present disclosure has the following beneficial effects. The user interests are modeled from the perspectives of dynamic interests and static interests, and multi-level dynamic and static interests are modeled based on long-term and short-term sequences, so that more accurate and practical user interest modeling is realized. The difference between a dynamic interest and a static interest lies in whether the interest remains stable for a period of time. The dynamic interest changes with time, while the static interest remains almost unchanged for a period of time. In addition to dynamic and static interests, the evolutionary interest changing from the static interest to the dynamic interest is taken into account, and more accurate personalized sequence recommendation is realized by adaptive fusion of the dynamic interest. the static interest and the evolutionary interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a model framework of an embodiment of a sequence recommendation method based on extracting and modeling of complex multi-mode user interests according to the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Aiming at the defects of the current sequence recommendation method in modeling user interests from the perspectives of long-term interests and short-term interests, the present disclosure designs a sequence recommendation method based on extracting and modeling of complex multi-mode user interests.

A sequence recommendation method based on extracting and modeling of complex multi-mode user interests designed by the present disclosure will be described in detail hereinafter, and the implementation process of the method is shown in FIG. 1.

The specific steps of the present disclosure are as follows.

In step (1), sequence division and vector embedding are performed. Specifically, a historical item interaction sequence H=(h₁, h₂, . . . , h_t) of a user is acquired, where h_i is an item corresponding to an i-th interaction behavior, a latest sequence H_l=(h_l¹, h_l², . . . , h_l^m) with a length of m is selected as a long-term sequence and a latest sequence H_s=(h_s¹, h_s², . . . , h_sⁿ) with a length of n is selected as a short-term sequence, where m>n is required. Based on a self-learning item embedding matrix F∈^k×d, items involved in the sequences are embedded to obtain a long-term embedding sequence F_l=(f_l¹, f_l², . . . , f_l^m) and a short-term embedding sequence F_s=(f_s¹, f_s², . . . , f_sⁿ), where k indicates a number of kinds of all items in all sequences, where F∈^m×d, F_s∈^n×d, f_lⁱ∈^d, f_sⁱ∈^d, and d indicates a vector embedding dimension. It is assumed that the user interaction sequence is (a₁, a₂, a₃, a₄, a₅, a₆), m=6, and n=3, a long-term sequence (a₁, a₂, a₃, a₄, a₅, a₆) and a short-term sequence (a₄, a₅, a₆) are obtained by splitting. The long-term embedding sequence F_l=(f_l^a1, f_l^a2, . . . , f_l^a6) and the short-term embedding sequence F_s=(f_s^a4, f_s^a5, f_s^a6) are obtained by embedding.

In step (2), embedding vectors of items are updated based on the multi-head self-attention mechanism. Specifically, the long-term embedding sequence F_l and the short-term embedding sequence F_s are inputted into two independent multi-head self-attention modules, respectively. Specifically, as far as the long-term embedding sequence F_l is concerned. in order to learn the sequence relationship between items, a self-learning position vector pos_lⁱ∈^d is assigned to each item on the sequence, and the item vector is updated to g_lⁱ=f_lⁱ+pos_lⁱ, so as to obtain the long-term embedding sequence G_l=(g_l¹, g_l², . . . , g_l^m) after being updated in position. The embedding sequence combined with the position vector is learned by the multi-head self-attention mechanism. The multi-head mechanism can model different information from different spaces, thus improving the representation ability of the model. Each attention head performs independent self-attention learning. Specifically, for the j-th attention head, the following three matrices are obtained by G_l:

A^j=G_lW_A^j,B^j=G_lW_B^j,C^j=G_lW_C^j,

where W_A^j∈^d×d′, W_B^j∈^d×d′ and W_C^j∈^d×d′ are three parameter matrices. Attention operation is performed to obtain the embedding sequence updated under the attention head:

${\hat{g}}_l^j=\mathrm{softmax}\left(\frac{A^j{B}^{j^T}}{\sqrt{d/0}}\right)C^i,$

where o indicates the number of attention heads. The updated long-term embedding sequence Ê_l=(ĝ_l¹, ĝ_l², . . . , ĝ_l^m)=(ê_l¹, ê_l², . . . , ê_l^m) is obtained by concatenating ĝ_l^j∈^m×d′ obtained from o attention heads, where Ê_l∈^m×(o×d′), ê_lⁱ∈^(o×d′). Similarly, the updated short-term embedding sequence Ê_s=(ê_s¹, ê_s², . . . , ê_sⁿ) is obtained by the multi-head attention mechanism. For the example in Step (1), the updated long-term embedding sequence Ê_l=(ê_l^a1, ê_l^a2, . . . , ê_l^a6) and the updated short-term embedding sequence Ê_s=(ê_s^a4, ê_s^a5, ê_s^a6) are obtained.

In step (3), initial dynamic and static interest modeling are performed. Specifically, an embedding vector ê_l^m of a last item in the long-term embedding sequence Ê_l is taken as a long-term dynamic interest p_l^d of a user. An attention weight of each vector in the long-term sequence to ê_l^m is calculated:

${\alpha }_i=\frac{\exp \left({\hat{e}}_l^{m^T}\mathrm{Re}\mathrm{LU}\left(W_l{\hat{e}}_l^i\right)\right)}{{\Sigma }_{j\in \left\{1,2,\dots ,m\right\}}\exp \left({\hat{e}}_l^{m^T}\mathrm{Re}\mathrm{LU}\left(W_l{ê}_l^j\right)\right)}$

where W_l∈^{(o×d′)×(o×d′)} is the parameter to be trained, ReLU is an activation function, and the long-term static interest is obtained through weighted summation based on the obtained weights:

p_l^x=Σ_i=1^mα_iê_lⁱ.

Similarly, the short-term dynamic interest p_s^d and the short-term static interest p_s^x are obtained based on the short-term embedding sequence Ê_s. Specifically,

$\begin{array}{c}{p}_s^d={ê}_s^n,\\ p_s^x={\sum }_{i=1}^m{\beta }_i{ê}_s^i,\\ {\beta }_i=\frac{\exp \left({ê}_s^{m^T}\mathrm{Re}\mathrm{LU}\left(W_s{ê}_s^i\right)\right)}{{\Sigma }_{j\in \left\{1,2,\dots ,n\right\}}\exp \left({\hat{e}}_s^{m^T}\mathrm{Re}\mathrm{LU}\left(W_s{\hat{e}}_s^j\right)\right)}.\end{array}$

For the example in step (2), the long-term dynamic interest is p_l^d=ê_l^a6, and the short-term dynamic interest is p_s^d=ê_s^a6. Attention weights of (ê_l^a1, ê_l^a2, . . . , ê_l^a6) to f_l^a6 are calculated, and then weighted summation is carried out to obtain a long-term static interest. Attention weights of (ê_s^a4, ê_s^a5, ê_s^a6) to f_s^a6 are calculated, and then weighted summation is carried out to obtain a short-term static interest.

In step (4), initial evolutionary interest learning is performed. Specifically, the long-term dynamic interest p_l^d and the long-term static interest p_l^x are concatenated, and then nonlinear transformation is performed to obtain a long-term evolutionary interest of the user:

p _l ^y=ReLU(W_l^y(p_l^d∥p_l^x)).

W_l^y∈^{(o×d′)×(2×o×d′)} is the parameter to be trained. Similarly, the short-term evolutionary interest p_s^y of the user is obtained based on the short-term dynamic interest p_s^d and the short-term static interest p_s^x.

In step (5), the dynamic interest of the user is obtained by element-wise summation of the long-term static interest p_l^d and the short-term static interest p_s^d:

p ^d =p _l ^d ⊕p _s ^d,

where ⊕ indicates element-wise addition. Similarly, the static interest p^x of the user is obtained by element-wise summation of the long-term static interest p_l^x and the short-term static interest p_s^x; and the evolutionary interest p^y of the user is obtained by element-wise summation of the long-term evolutionary interest p_l^y and the short-term evolutionary interest p_s^y.

In step (6), interest fusion is performed. Specifically, attention weights of the dynamic interest p^d, the static interest p^x and the evolutionary interest p^y to the embedding vector ê_l^m of the last item are calculated, which are specifically defined as:

$\begin{array}{c}{\alpha }_p=\frac{\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^d\right)\right)}{\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^d\right)\right)+\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^x\right)\right)+\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^y\right)\right)}\\ {\alpha }_x=\frac{\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^x\right)\right)}{\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^d\right)\right)+\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^x\right)\right)+\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^y\right)\right)}\\ {\alpha }_y=\frac{\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^y\right)\right)}{\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^d\right)\right)+\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^x\right)\right)+\exp \left({ê}_l^m\mathrm{Re}\mathrm{LU}\left(W_f{p}^y\right)\right)}\end{array}$

where weighted summation and transformation are carried out to obtain the fused user interest p∈^d:

p=W _r(α_pp^d+α_xp^x+α_yp^y)

where W_r∈^d×(o×d′) is a parameter to be trained.

In step (7), recommendation is performed. Specifically, a product of p with embedding F of each item is calculated as a recommendation score of each item, and top items with highest scores are recommended for the user.

Claims

1. A sequence recommendation method based on extracting and modeling of complex multi-mode user interests, comprising: Step 1, acquiring a historical item interaction sequence of a user, and obtaining a long-term embedding sequence Fl=(fl1, fl2, . . . , flm) and a short-term embedding sequence Fs=(fs1, fs2, . . . , fsn) through a self-learning item embedding matrix, where m>n;Step 2, inputting the long-term embedding sequence and the short-term embedding sequence into two independent multi-head self-attention modules, respectively, to obtain an updated long-term embedding sequence and an updated short-term embedding sequence;Step 3, with an embedding vector of a last item in the updated long-term embedding sequence as a long-term dynamic interest of the user, calculating attention weights of the updated long-term embedding sequence to the embedding vector of the last item, and performing weighted summation to obtain a long-term static interest of the user, and similarly, obtaining a short-term dynamic interest and a short-term static interest based on the updated short-term embedding sequence, wherein:the long-term dynamic interest pld of the user is the embedding vector êlm of the last item in the updated long-term embedding sequence Êl=(êl1, êl2, . . . , êlm), that is, the long-term dynamic interest is pld=êlm, and the long-term static interest is defined as:

2. The method according to claim 1, wherein in Step 1, the historical item interaction sequence of the user is expressed as H=(h1, h2, . . . , ht), where hi is an item corresponding to an i-th interaction behavior; a long-term sequence is a latest sequence Hl=(hl1, hl2, . . . , hlm) with a length of m, and a short-term sequence is a latest sequence Hs=(hs1, hs2, . . . , hsn ) with a length of n; based on a self-learning item embedding matrix F∈k×d, items involved in the sequences are embedded to obtain a long-term embedding sequence Fl=(fl1, fl2, . . . , flm) and a short-term embedding sequence Fs=(fs1, fs2, . . . , fsn), where k indicates a number of kinds of all items in all sequences, where Fl∈m×d, Fs∈n×d, fli∈d, fsi∈d, and d indicates a vector embedding dimension.

3. The method according to claim 2, wherein in Step 2, the updated long-term embedding sequence Êl is defined as: Êl=(ĝl1,ĝl2, . . . , ĝlm)=(êl1,êl2, . . . , êlm)where êli is an embedding representation of a corresponding item i after being updated by an attention mechanism, and ĝlj∈m×d′ is an embedding sequence obtained by a j-th attention head, which is defined as:

4. The method according to claim 3, wherein in Step 4, the long-term evolutionary interest of the user is defined as: ply=ReLU(Wly(pld∥plx))where pld is the long-term dynamic interest, plx is the long-term static interest, (⋅|⋅) is a concatenation operation, and Wly∈(o×d′)×(2×o×d′) is a parameter to be trained;the short-term evolutionary interest of the user is defined as: psy=ReLU(Wsx(psd∥psx)).

5. The method according to claim 4, wherein in Step 5, the dynamic interest pd of the user is defined as: pd=pld⊕psd,where ⊕ indicates element-wise addition, pld is the long-term dynamic interest, and psd is the short-term dynamic interest;the static interest px of the user is defined as: px=plx⊕psx,where ⊕ indicates element-wise addition, plx is the long-term static interest, and psx is the short-term static interest;the evolutionary interest py of the user is defined as: py=ply⊕psy,where ⊕ indicates element-wise addition, ply is the long-term evolutionary interest, and psy is the short-term evolutionary interest.

6. The method according to claim 5, wherein in Step 6, the fused user interest is defined as: