STACKING-ENSEMBLE-BASED APT ORGANIZATION IDENTIFICATION METHOD AND SYSTEM, AND STORAGE MEDIUM

Abstract

An APT organization identification method, system and storage medium based on a stacking ensemble are provided, the method comprising: using a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set; based on the malicious behavior vector feature set, calculating correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data; constructing a multi-model fusion stacking ensemble, learning an APT organization identification model, using the APT organization identification model to perform an identification on new ATP attacks. The feature selection of high-dimensional behavior vector features reduces the complexity of the data set; the imbalance of samples in the data set is also considered, and multi-model integrated training to improve the recognition accuracy is adopted; in addition, the APT organization identification model for malicious samples is obtained through machine learning training, which improves the automatic identification efficiency of new sample is improved.

Description

TECHNICAL FIELD

The invention belongs to the technical field of network security, and in particular relates to an APT organization identification method, system and storage medium based on a stacking ensemble.

TECHNICAL BACKGROUND

Advanced Persistent Threat, APT, is an attack form that uses advanced attack methods to carry out long-term persistent network attacks on specific targets. Different from traditional network attacks, APT attacks have the characteristics of concealment, target, persistence and organization etc. The attack methods are varied. The attack effect is remarkable and difficult to prevent. Cyberattacks carried out by APT organizations usually have political or economic purposes, have a huge impact on countries and enterprises, and pose an increasingly serious threat to various high-level information security systems. Performing organization differentiation of malware samples that implement APT attacks is beneficial to trace real attacking organization entities and to better distinguish and identify specific sources of attack activities. In addition, APT organization identification of malware is a method to detect APT attacks.

APT organization identification methods differ between academia and industry. In academia, the main method relies on similarity analysis of malicious code. For example, Qiao, Yanchen et al. proposed an automatic malware homology identification method based on API calls. The method obtains the API set of malicious samples through static analysis, and then uses the Jaccard similarity coefficient to calculate the homology degree of different malware based on the six calling behaviors defined by programming habits, and establishes a threshold through experience to compare with the homology degree to draw a conclusion of whether the samples are similar. This method may be used to judge the homology between APT samples and determine the organization to which they belong. Hu Zhigiang et al. performed a reverse analysis on malicious sample files to obtain functions of the files, clustering the functions according to the similarity of the functions to obtain a feature set, and set a threshold, and classify malicious sample files whose number of common categories reaches the threshold into one category and mark the same APT organization logo for each type of classified malicious sample files. Chen, W, et al. proposed a new genetic model incorporating knowledge graphs of malware behavior. The method first establishes a genetic model based on node contents, and extracts gene sequences of all malware belonging to each APT organization, then calculates the similarity between the malware and the gene bank, and judges which APT organization the malware belongs to according to similarity scores.

In the industry, the identification of APT organizations is more inclined to the correlation analysis of malicious code structure and its attack chain. For example, FireEye Laboratory analyzed 11 APT advanced persistent attacks in 2013, and found the same code segments, timestamps, digital certificates etc. in the malicious codes used in the attacks, and believed that the attacks were all carried out by the same group. Venustech used shellcode functions and code similarity of some vulnerability samples as the characteristics of correlation analysis, and then traced the source to Hedwig organization.

The above-mentioned industrial analysis techniques are mainly based on manual analysis by relevant security experts, which are greatly influenced by the experience of experts. Secondly, they cannot meet the needs of a large number of samples, with low efficiency and time-consuming. The static API function features that the automatic identification technology in academia relies on will make feature extraction difficult due to the obfuscation and packing techniques of malware. In addition, the current methods mainly rely on known malicious code samples. If variants are identified based only on existing samples, it may lead to inefficient or even ineffective identification.

SUMMARY OF THE INVENTION

The main objective of the present invention is to overcome the shortcomings and deficiencies of the current technology, and to provide an APT organization identification method, system and storage medium based on a stacking ensemble, which may more accurately identify the APT organization to which an APT attack behavior belongs.

In order to achieve the above objective, the present invention adopts the following technical solutions:

The present invention provides an APT organization identification method based on a stacking ensemble, characterized in that, comprising the following steps:

using a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set;

based on the malicious behavior vector feature set and an APT organization tag, calculating correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;

constructing a multi-model fusion stacking ensemble, learning an APT organization identification model, using the APT organization identification model to perform an organization identification on new ATP attacks.

As a preferred technical solution, using the TF-IDF algorithm combined with the n-gram to extract and vectorize the behavior features from the malware samples to form a behavior data set is specifically:

for behavior text features of the malware samples, first generating n-gram texts, the n-gram may be selected according to actual data, it is recommended to choose n-gram=(1,5), then counting a text frequency TF of each text separately, then attaching a weight parameter IDF to each text;

${\mathrm{TF}}_{i,j}=\frac{n_{i,j}}{{\sum }_k{n}_{k,j}}$

wherein TF_i,j: a frequency of a texti in a sample j; n_i,j: a number of times the text i appears in the sample j; Σ_kn_k,j: a total number of text appearing in the sample j;

then calculating the weight parameters:

${\mathrm{IDF}}_{i,j}=\log \frac{❘D❘}{❘j:i\in d_j❘+1}$

wherein |D| represents a total number of samples, |j:i∈d_k| represents a number of samples containing the text i, in order to prevent a denominator from being zero, 1 is added, a final weight calculation formula of each text is:

TF−IDF _i,j =TF _i,j ×IDF _i,j

through a TF-IDF method for calculating malicious sample behavior feature text combined with the n-gram, preprocessing data, calculating text frequency features, performing a feature vectorization on behavior text data to form a semantic matrix, forming the malicious behavior vector feature set.

As a preferred technical solution, feature data extracted by the n-gram combined with a TF-IDF method comprises more feature attributes, first, performing a first primary selection of the malicious behavior vector feature set, calculating correlations between features and features, filtering out features with information redundancy between features.

As a preferred technical solution, calculating the correlations between features and features is specifically:

input: a behavior vector dataset F, a number of features F_n, thresholds ε₁, ε₂;

randomly selecting a feature X₁, calculating its information entropy H(X₁), if H(X₁)>ε₁ is satisfied, then adding it to a feature set to be selected S, otherwise, continue to select;

for i=2, . . . , F_n

calculating an information entropy of the feature X_i, if H(X_i)>ε₁ is satisfied, then judging correlations between this feature and all other features X_j in S:

${\rho }_{X_i,X_j}=\frac{\mathrm{cov}\left(X_i,X_j\right)}{{\sigma }_{X_i}{\sigma }_{X_j}}=\frac{E(\left(X_i-{\mu }_{x_i}\right)\left(X_j-{\mu }_{x_j}\right)}{{\sigma }_{X_i}{\sigma }_{X_j}}$

wherein cov(X_i,X_j) is a covariance between X_i and X_j, σ_Xi is a variance of X_i, σ_xj is a variance of X_j, μ_xi is a mean of X_i, μ_xj is a mean of X_j, and E is an average value;

if |ρ_Xi,Xj|>ε₂, then adding the feature X_i to the feature set to be selected S;

if |ρ_Xi,Xj|>ε₂, calculating the information entropy of X_i and X_j, if H(X_j)>H(X_i), then replacing the feature X_i with X_j in S;

outputting feature subset S.

As a preferred technical solution, in a process of calculating the correlations between features and features, each feature selected in the feature subset S has more information, and maintaining a low correlation between all features, values of ε₁ and ε₂ need to be adjusted according to experimental effects by adjusting parameters, and need to satisfy:

${\epsilon }_1<\frac{1}{n}\sum _{X_i\in D}H\left(X_i\right),{\epsilon }_2>\frac{1}{n^2}\sum _{X_i,X_j\in D}\rho \left(X_i,X_j\right)$

then, on the obtained feature subset S, further using a chi-square test to evaluate correlations between the features and category labels, and selecting top N features in each category with the chi-square values.

As a preferred technical solution, calculating the chi-square values between the features and the categories is specifically:

for S_m categories in a feature subset S, calculating a chi-square value of each feature in each category, and arranging the features in a descending order of the chi-square values;

from feature sets of each category, selecting top N feature text and putting them into a new feature subset S′;

keeping one of the repeated features in S′, and deleting the rest;

outputting the new feature subset S′, and the algorithm ends.

As a preferred technical solution, constructing the multi-model fusion stacking ensemble, learning the APT organization identification model is specifically:

based on a new feature subset S′ after a feature selection and a corresponding labeled APT organization data as an original data set, a stacking ensemble classification model first divides the original data set into several data sets, which are input to each basic learner of a first-layer prediction model, each base learner outputs its prediction result; wherein the base learner selects a classification algorithm that has a good classification effect on unbalanced data, namely a support vector machine, a random forest, an Adaboost algorithm, then output data of a first layer is used as input data of a second layer, performing a training on a meta-learner of the second layer, and outputting a final prediction result by a model of the second layer, wherein a XGBoost algorithm is used as the meta-learner.

As a preferred technical solution, a specific training method of the stacking ensemble learning is: for a data set D={(x_n,y_n), n=1, . . . , N}, wherein x_n is a feature vector of a n-th sample, y_n is a predicted value corresponding to the n-th sample, m is a number of features included, that is, each feature vector is (x₁, x₂, . . . , x_m); randomly dividing data into K equal-sized subsets D₁, D₂, . . . , D_k; wherein D′=D−D_k, D′ and D_k are a training set and a test set in a k-fold cross-validation respectively, the training set D′ obtains models M_SVM, M_RF, M_Ada through three basic learners of the first layer; for each sample x_n in the test set D_k, a basic learning prediction for it is represented as T_kn; after completing a cross-validation process, output data of the three basic learners constitutes a new data sample D_new={(y_n, T_k1, . . . , T_kn), n=1, . . . , N}; using the data set D_new generated by the basic learners as the input data of the second layer of a stacking, then obtaining a model M_Xgb by training the XGBoost algorithm of the second layer.

The present invention further provides an APT organization identification system based on a stacking ensemble, comprising a feature extraction module, a feature screening module, and an identification module;

the feature extraction module is used to use a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set;

the feature screening module is used, based on the malicious behavior vector feature set, to calculate correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;

the identification module is used to construct a multi-model fusion stacking ensemble, learn an APT organization identification model, using the APT organization identification model to perform an organization identification on new ATP attacks.

The present invention further provides a storage medium storing a program, when the program is executed by a processor, the APT organization identification method based on a stacking ensemble is implemented.

Compared with the current technology, the present invention has the following advantages and beneficial effects:

(1) The present invention obtains behavior data of malware based on dynamic analysis and uses a TF-IDF algorithm combined with n-gram to extract and vectorize behavior features from malware samples, which may effectively deal with certain transformation techniques, such as obfuscation and packing etc., which improves the effectiveness of extracting the required feature data.

(2) In the present invention, the feature selection of high-dimensional behavior vector features reduces the complexity of the data set; the imbalance of samples in the data set is also considered, and multi-model integrated training to improve the recognition accuracy is adopted; in addition, the APT organization identification model for malicious samples in this patent is obtained through machine learning training, which improves the automatic identification efficiency of new samples.

(3) The present invention may find the organization to which an APT attack belongs based on an APT organization identification model, which is beneficial to trace the real attacking organization entity, and better distinguish and identify attack activities of specific sources.

DESCRIPTION OF FIGURES

FIG. 1 is a flow chart of an APT organization identification method based on a stacking ensemble of the present invention;

FIG. 2 is a model integration diagram of stacking of the present invention.

DETAILED DESCRIPTION

The present invention will be described in further detail below with reference to the embodiments and the accompanying figures, but the embodiments of the present invention are not limited thereto.

Embodiments

As shown in FIG. 1, an APT organization identification method based on a stacking ensemble of this embodiment, comprising the following steps:

S1: using a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples, the n-gram may be selected according to actual data, it is recommended to choose n-gram=(1.5), to form a malicious behavior vector feature set;

counting a text frequency (TF) of each text for the behavior text features of malicious samples, then attaching a weight parameter (IDF) to it.

${\mathrm{TF}}_{i,j}=\frac{n_{i,j}}{\sum _k{n}_{i,k}}$

wherein TF_i,j: a frequency of a text i in a sample j; n_i,j: a number of times the text i appears in the sample j; Σ_kn_k,j: a total number of text appearing in the sample j.

Then calculating the weight:

${\mathrm{IDF}}_{i,j}=\log \frac{❘D❘}{❘j:t_i\in d_j❘+1}$

wherein |D| represents a total number of samples, |j:i∈d_j| represents a number of samples containing the text i, in order to prevent a denominator from being zero, 1 is added, a final weight calculation formula of each text is:

TF−IDF _i,j =TF _i,j ×IDF _i,j

through a TF-IDF method for calculating malicious sample behavior feature text combined with the n-gram, preprocessing data, calculating text frequency features, performing a feature vectorization on behavior text data to form a semantic matrix, forming the malicious behavior vector feature set.

S2: based on the malicious behavior vector feature set, calculating correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;

Since feature data extracted by the n-gram method in step S1 comprises more feature attributes, it is necessary to, first, perform a first primary selection of the feature set, calculate correlations between features and features, filter out features with information redundancy between features.

Further, in this embodiment, calculating the correlations between features and features is specifically an algorithm that is:

input: a behavior vector dataset F, a number of features F_n, thresholds ε₁, ε₂.

output: feature subset S

(1) Randomly selecting a feature X₁, calculating its information entropy H(X₁), if H(X₁)>ε₁ is satisfied, then adding it to a feature set to be selected S. Otherwise, continue to select.

(2) For i=2, . . . , F_n

(3) Calculating an information entropy of the feature X_i, if H(X_i)>ε₁ is satisfied, then judging correlations between this feature and all other features X_i in

$S:{\rho }_{X_i,X_j}=\frac{\mathrm{cov}\left(X_i,X_j\right)}{{\sigma }_{X_i}{\sigma }_{X_j}}=\frac{E(\left(X_i-{\mu }_{x_i}\right)\left(X_j-{\mu }_{x_j}\right)}{{\sigma }_{X_i}{\sigma }_{x_j}},$

wherein cov(X_i,X_j) is a covariance between X_i and X_j, σ_Xi is a variance of X_i, σ_xj is a mean of X_j, μ_xi is a mean of X_i, and E is an average value.

(4) If |ρ_Xi,Xj|<ε₂, then adding the feature X_i to the feature set to be selected S. If |ρ_Xi,Xj|>ε₂, calculating the information entropy of X_i and X_j, if H(X_j)>H(X_i), then replacing the feature X_i with X_j in S.

(5) Outputting the feature subset S.

Further, in the above feature selection process based on correlation, each feature selected in the feature subset S has more information, and maintains a low correlation between all features. Values of ε₁ and ε₂ need to be adjusted according to experimental effects by adjusting parameters, and need to satisfy:

${\epsilon }_1<\frac{1}{n}\sum _{X_i\in D}H\left(X_i\right),{\epsilon }_2>\frac{1}{n^2}\sum _{X_i,X_j\in D}\rho \left(X_i,X_j\right)$

then, on the obtained simplified feature subset S, further using a chi-square test to evaluate correlations between the features and category labels, and selecting top N features in each category with the chi-square values.

In the present embodiment, the feature primary selection algorithm based on a chi-square test:

Input: feature subset S, number of categories S_m, feature selection dimension N

Output: new feature subset S′

(1) for S_m categories in a feature subset S, calculating a chi-square value of each feature in each category, and arranging the features in a descending order of the chi-square values.

a) from feature sets of each category, selecting top N feature text and putting them into a feature subset S′;

b) keeping one of the repeated features in S′, and deleting the rest;

c) outputting the new feature subset S′, and the algorithm ends.

In the feature selection part of the method of this embodiment, firstly, the non-redundant features containing the most information are selected by using the information entropy and the correlation degree, then the chi-square test is further used to select the feature subset with high correlation with the category, which realizes a reduction of data dimension through a feature selection, and a better effective feature set is obtained at the same time.

S3. constructing a multi-model fusion stacking ensemble, learning an APT organization identification model, using the APT organization identification model to perform an organization identification on new ATP attacks.

As shown in FIG. 2, this embodiment is based on the feature subset S′ after a feature selection and a corresponding labeled APT organization data as an original data set, a stacking ensemble classification model first divides the original data set into several data sets, which are input to each basic learner of a first-layer prediction model, each base learner outputs its prediction result. Wherein the base learner selects a classification algorithm that has a good classification effect on unbalanced data, namely a support vector machine, a random forest, an Adaboost algorithm, then output data of a first layer is used as input data of a second layer, performing a training on a meta-learner of the second layer, and outputting a final prediction result by a model of the second layer, wherein a XGBoost algorithm is used as the meta-learner.

Further, a specific training method of the stacking ensemble learning is: for a data set D={(x_n,y_n), n=1, . . . , N}, wherein x_n is a feature vector of a n-th sample, y_n is a predicted value corresponding to the n-th sample, m is a number of features included, that is, each feature vector is (x₁, x₂, . . . , x_m). Randomly dividing data into K equal-sized subsets D₁, D₂, . . . , D_k. Wherein D′=D−D_k, D′ and D_k are a training set and a test set in a k-fold cross-validation respectively, the training set D′ obtains models M_SVM, M_RF, M_Ada through three basic learners of the first layer. For each sample x_n in the test set D_k, a basic learning prediction for it is represented as T_kn. After completing a cross-validation process, output data of the three basic learners constitutes a new data sample D_new={(y_n, T_k1, . . . , T_kn), n=1, . . . , N}. Using the data set D_new generated by the basic learners as the input data of the second layer of a stacking, then obtaining a model M_Xgb by training the XGBoost algorithm of the second layer.

In another embodiment, there is provided an APT organization identification system based on a stacking ensemble, the system comprises a feature extraction module, a feature screening module and an identification module;

the feature extraction module is used to use a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set;

the feature screening module is used, based on the malicious behavior vector feature set, to calculate correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;

the identification module is used to construct a multi-model fusion stacking ensemble, learn an APT organization identification model, using the APT organization identification model to perform an identification on new ATP attacks.

It should be noted here that the system provided by the above-mentioned embodiments is only illustrated by the division of the above-mentioned functional modules. In practical applications, the above functions may be assigned to different functional modules according to the needs, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. In addition, the algorithms used in the stacking ensemble are not limited to the few algorithms in the example.

In another embodiment of the present application, a storage medium is also provided. The storage is with a program. When the program is executed by a processor, the APT organization identification method based on a stacking ensemble is specifically:

using a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set;

based on the malicious behavior vector feature set and, calculating correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;

constructing a multi-model fusion stacking ensemble, learning an APT organization identification model, using the APT organization identification model to perform an identification on new ATP attacks.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it may be implemented by any one or a combination of the following techniques known in the art: discrete logic circuits with logic gates for implementing logic functions on data signals, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA) etc.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, simplification should be equivalent replacement manners, which are all included in the protection scope of the present invention

Claims

1. An APT organization identification method based on a stacking ensemble, characterized in that, comprising the following steps: using a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set;based on the malicious behavior vector feature set and an APT organization tag, calculating correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;constructing a multi-model fusion stacking ensemble, learning an APT organization identification model, using the APT organization identification model to perform an organization identification on new ATP attacks.

2. The APT organization identification method based on a stacking ensemble according to claim 1, characterized in that, using the TF-IDF algorithm combined with the n-gram to extract and vectorize the behavior features from the malware samples to form a behavior data set is specifically: for behavior text features of the malware samples, first generating n-gram texts, then counting a text frequency TF of each text separately, then attaching a weight parameter IDF to each text;

3. The APT organization identification method based on a stacking ensemble according to claim 1, characterized in that, feature data extracted by the n-gram combined with a TF-IDF method comprises more feature attributes, first, performing a first primary selection of the malicious behavior vector feature set, calculating correlations between features and features, filtering out features with information redundancy between features.

4. The APT organization identification method based on a stacking ensemble according to claim 3, characterized in that, calculating the correlations between features and features is specifically: input: a behavior vector dataset F, a number of features Fn, thresholds ε1, ε2;randomly selecting a feature X1, calculating its information entropy H(X1), if H(X1)>ε1 is satisfied, then adding it to a feature set to be selected S, otherwise, continue to select;for i=2, . . . , Fn calculating an information entropy of the feature Xi, if H(Xi)>ε1 satisfied, then judging correlations between this feature and all other features Xj in S:

5. The APT organization identification method based on a stacking ensemble according to claim 4, characterized in that, in a process of calculating the correlations between features and features, each feature selected in the feature subset S has more information, and maintaining a low correlation between all features, values of ε1 and ε2 need to be adjusted according to experimental effects by adjusting parameters, and need to satisfy:

6. The APT organization identification method based on a stacking ensemble according to claim 1, characterized in that, calculating of the chi-square values between the features and the categories is specifically: for Sm categories in a feature subset S, calculating a chi-square value of each feature in each category, and arranging the features in a descending order of the chi-square values;from feature sets of each category, selecting top N feature text and put them into a new feature subset S′;keeping one of the repeated features in S′, and deleting the rest;outputting the new feature subset S′, and the algorithm ends.

7. The APT organization identification method based on a stacking ensemble according to claim 1, characterized in that, constructing the multi-model fusion stacking ensemble, learning the APT organization identification model is specifically: based on a new feature subset S′ after a feature selection and a corresponding labeled APT organization data as an original data set, a stacking ensemble classification model first divides the original data set into several data sets, which are input to each basic learner of a first-layer prediction model, each base learner outputs its own prediction result; wherein the base learner selects a classification algorithm that has a good classification effect on unbalanced data, namely a support vector machine, a random forest, an Adaboost algorithm, then output data of a first layer is used as input data of a second layer, performing a training on a meta-learner of the second layer, and outputting a final prediction result by a model of the second layer, wherein an XGBoost algorithm is used as the meta-learner.

8. The APT organization identification method based on a stacking ensemble according to claim 7, characterized in that, a specific training method of the stacking ensemble learning is: for a data set D={(xn,yn), n=1, . . . , N}, wherein xn is a feature vector of a n-th sample, yn is a predicted value corresponding to the n-th sample, m is a number of features included, that is, each feature vector is (x1, x2, . . . , xm); randomly dividing data into K equal-sized subsets D1, D2, . . . , Dk; wherein D′=D−Dk, D′ and Dk are a training set and a test set in a k-fold cross-validation respectively, the training set D′ obtains models MSVM, MRF, MAda through three basic learners of the first layer; for each sample xn in the test set Dk, a basic learning prediction for it is represented as Tkn; after completing a cross-validation process, output data of the three basic learners constitute a new data sample Dnew={(yn, Tk1, . . . , Tkn), n=1, . . . , N}; using the data set dnew generated by the basic learners as the input data of the second layer of a stacking, then obtaining a model MXgb by training the XGBoost algorithm of the second layer.

9. An APT organization identification system based on a stacking ensemble, characterized in that, comprising a feature extraction module, a feature screening module and an identification module; the feature extraction module is used to use a TF-IDF algorithm combined with an n-gram to extract and vectorize behavior features from malware samples to form a malicious behavior vector feature set;the feature screening module is used, based on the malicious behavior vector feature set, to calculate correlations between features and chi-square values between the features and categories, performing screening twice on the malicious behavior vector feature set to obtain an improved low-dimensional feature subset data;the identification module is used to construct a multi-model fusion stacking ensemble, learn an APT organization identification model, using the APT organization identification model to perform an identification on new ATP attacks.

10. A storage medium storing a program, characterized in that, when the program is executed by a processor, the APT organization identification method based on a stacking ensemble according to claim 1 is implemented.