Patent Application
20230161688
The present application claims priority to Chinese Patent Application No. 202111405653.4, filed on Nov. 24, 2021, the content of which is incorporated herein by reference in its entirety.
The present disclosure relates to the field of automatic driving simulation training, in particular to an intelligent test method for dynamically generating a test case according to test performance of a tested system.
Compared with the training in real scenes, simulation is more efficient and low-cost, so it is widely used in the process of training system. Case generation, as an important link, can facilitate training of various situations that the tested system may encounter in the simulation environment, and simulating unusual special situations to improve the training efficiency. In order to test the performance of the system under test in various scenarios, the existing case generation methods cover various real situations by generating a large number of cases, but this method requires higher requirements for computing power and time cost. How to dynamically generate effective cases more intelligently according to the existing performance of an automatic training system is an urgent problem to be solved.
In view of the shortcomings of the prior art, the present disclosure provides an intelligent test method for dynamically generating test cases according to the test performance of a tested system. The method takes into account the performance of the trained agent in the previous test cases in the simulation, and dynamically generates the cases that are more in line with the test requirements according to the test results.
The purpose of the present disclosure is realized by the following technical solution:
An intelligent test method for dynamically generating test cases according to the test performance of a tested system comprises following steps:
S1, selecting a test template test_case_template containing N variables var.
S2, sampling values in a value range of each variable var, and forming a sampling set Q with all of the sampled values.
S3, selecting a value from a sampling set Q corresponding to each variable to form a template variable numerical sampling vector g=(s1, s2, . . . , sN), s1∈Q1, s2∈Q2, . . . , sN∈QN, and then generating a set G_init from g.
S4, starting a kth round of test, wherein when k=1, the set Gk=G_init; traversing each template variable numerical sampling vector in the set Gk, obtaining a category label(g) corresponding to each template variable numerical sampling vector g according to the test template when traversing, and obtaining a test performance data set Dk corresponding to the set Gk.
S5, determining whether k is equal to 1. If k=1, training a decision tree Tk whose classification error rate is less than a preset threshold epsilonk by the test performance data set Dk, predicting the label(g) by the decision tree Tk according to any g; and putting a sample with wrong classification into a data set Dk+1. If k>1, classifying and verifying the samples in the test performance data set Dk by using decision trees T1, T2, . . . , Tk−1, and carrying out weighted average on a classification result of each sample for each decision tree to calculate an error rate. If the error rate is greater than or equal to a preset threshold epsilonk−1, training a decision tree Tk by the test performance data set Dk, predicting the label(g) by the decision tree Tk according to any g, putting the samples with wrong classification into the data set Dk+1 and going to S6. If the error rate is less than the preset threshold epsilonk−1, ending testing the test template test_case_template;
S6, finding all areas in which the classification label of g is predicted to be the first class by Tk in a N-dimensional space where g is located according to all leaf nodes classified as a first class in Tk, and taking a union of the areas, wherein the union is denoted as Rk;
S7, sampling a batch of new vectors g_new in the N-dimensional space where g is located, keeping a vector distribution density positively correlated with a distance between g_new and a boundary of Rk; forming a set G_new by g_new, letting k=k+1, if k is less than a threshold of preset rounds threshold, letting Gk=G_new, going to S4, and starting a next round of testing; otherwise, ending testing the test template test_case_template.
Further, in S4, in order to obtain the data set Dk corresponding to the set Gk, first, a copy of the test template test_case_template is generated, the vari in the copy is replaced by si in g, and the replaced copy of the test template is defined as a test case and denoted as H(g); the tested system is tested with H(g) to obtain a test score score(g), and then a two-class category label(g) for the test case H(g) is provided according to the score(g), and finally, a binary group (g, label(g)) is put into the test performance data set Dk.
Further, the step of keeping a vector distribution density positively correlated with a distance between g_new and a boundary of the area R in S7 is specifically that the vector in G_new satisfying the following conditions:
(1) The closer to the boundary of Rk is, the denser the vector distribution is.
(2) A certain distance exists between a newly sampled vector and a tested vector.
Further, the distance between the newly sampled vector g_new and the tested vector g_old satisfies:
An absolute value of projection of g_new−g_old in the j dimension is greater than or equal to djk, where g_old is an arbitrary vector belonging to the union of G1, G2, . . . , Gk, and djk is the threshold of the distance between g_new and g_old in the j dimension in the kth round.
Further, the category label(g) is LOW or NORMAL, the category label(g) is LOW when the test score(g) is less than a preset score threshold, and the category label(g) is NORMAL when the test score(g) is greater than or equal to the preset score threshold.
Further, djk satisfies the following condition:
d
jk=alpha*djk-1
where alpha is a given normal number, 0<alpha<1.
The present disclosure has the following beneficial effects:
In the simulation process, the present disclosure makes full use of the results of previous test cases, and by constructing a plurality of decision trees, the algorithm can accurately predict the performances of trained agents in different cases, and learn the spatial division of variables that will lead to different results in the simulation test. According to this spatial division, the case distribution characteristics of the next stage are set to improve the effectiveness of case generation and training efficiency. In addition, the whole process can be iterated online in real time, and the update of the decision tree is completed in parallel with the case test. At this time, this method can significantly improve the efficiency of simulation training, and it is universal and suitable for simulation case testing in various scenarios, and excellent performance can be obtained in many application scenarios that need virtual simulation training.
The purpose and effect of the present disclosure will become more apparent from description of the present disclosure in detail according to the following drawings and preferred embodiments. It should be understood that the specific embodiments described here are only for explaining the present disclosure, but not for limiting the present disclosure.
The method of the present disclosure as shown in
S1, among the cases that are prone to accidents in automatic driving, a test template case test_case_template was selected for the tested automatic driving agent vehicle to overtake the preceding vehicle. The test template contained six variables Var={A_x, A_y, A_v, B_x, B_y, B_v}, where (A_x, A_y) is the current coordinate of the tested agent vehicle with a unit of m; A_v is the current speed of the measured vehicle with a unit of km/h; (B_x, B_y) is the coordinate unit of the preceding vehicle of a unit of m; B_v is the speed of the preceding vehicle with a unit of km/h.
Each variable has a definite value range: A_x belongs to [2,4], A_y belongs to [0,3], B_x belongs to [1,5], B_y belongs to [4,10], A_v belongs to [30,50] and B_v belongs to [30,60].
S2, values were sampled for 100 times in the value range for each variable var, variables were evenly distributed in the value range thereof during sampling, and a sampling set Q is formed with all the sampled values of each variable var.
S3, a value was selected from a sampling set Q corresponding to each variable; a vector g was obtained after each time of sampling, and then a set G_init={g1, g2, . . . g100} was generated from g, where gi is called a template variable numerical sampling vector, and the value of the jth dimension of gi corresponds to the numerical sampling of the jth variable.
S4, the kth round of test was started. When k=1, the set Gk=G_init; dk is the threshold value of the distance between g and g_old in the kth round, d1=0.1. When k>1, dk=0.9dk−1, and the preset threshold of the error rate epsilon=0.1.
Each template variable numerical sampling vector in the set Gk was traversed, and the category label label(g) corresponding to each template variable numerical sampling vector g was obtained according to the test template in the traversal process, and the test performance data set Dk corresponding to the set Gk was obtained. The specific operation was as follows:
(1) A copy of test_case_template of autopilot was generated, vari in the copy was replaced with si in G. The test template copy after replacement is called as a test case and denoted as H(g).
(2) The tested system was tested with H(g) to get a test score score(g), and then a two-class category label(g) was provided for the test case H(g) according to the score(g). In an embodiment, the category label(g) was LOW or NORMAL. When the test score(g) was less than the preset score threshold, the category label(g) was LOW, and when the test score(g) was greater than or equal to the preset score threshold, the category label(g) was NORMAL.
(3) The binary group (g, label(g)) was put into the test performance data set Dk.
In the embodiment of automatic driving, the ith dimension numerical sampling gij in gi was used to replace the ith template variable vari in the test template copy, where i=1, 2, . . . , N. The test template copy after replacement is called an automatic driving test case and denoted as H(gi), where H represents the mapping relationship from the template variable numerical sampling vector to the generated test case. The system under test was tested by the automatic driving test case H(gi), and the test score of the agent unmanned vehicle was score(gi). Then, a test result label label(gi) was set for the test case H(g) according to score(gi). The rules for setting the classification label were as follows: if score(gi) was less than 60, then label(gi) was set to low performance LOW, and if score(gi) was in a range of [60,100], then label(gi) was set as normal performance NORMAL, and the binary group (gi, label(gi)) was put into the test performance data set Dk.
S5, whether k is equal to 1 was determined. If k=1, Dk was used to train a decision tree Tk whose classification error rate is less than the preset threshold epsilonk, the label(g) was predicted by the decision tree Tk according to any g, and sample with wrong classification was put into the data set Dk+1. If k>1, decision trees T1, T2, . . . , Tk−1 were used to classify and verify the samples in Dk, and weighted average was carried out on a classification result of each sample for each decision tree to calculate an error rate. If the error rate was greater than or equal to a preset threshold epsilonk−1, a decision tree Tk would be trained by Dk, the label(g) would be predicted by Tk according to any g, the samples with wrong classification would be put into the data set Dk+1 and then S6 would be executed. If the classification error rate was less than the preset threshold epsilonk−1, the test of the test template test_case_templat would be ended.
S6, according to all leaf nodes classified as a first class in Tk, all areas in which the classification label of g was predicted to be the first class by Tk are found in the N-dimensional space where g was located, and a union of the areas was taken. The union was denoted as Rk.
In this embodiment of automatic driving, all the leaf nodes classified as LOW in Tk are found in the N-dimensional space where G is located, and all the areas predicted by Tk as LOW by the category label of G are merged and marked as Rk. The category label of any vector outside Rk is NORMAL.
S7, a batch of new vectors g_new was sampled in the N-dimensional space where g is located, and a vector distribution density was kept positively correlated with a distance between g_new and a boundary of Rk; g_new formed a set G_new. Let k=k+1, and if k was less than a threshold of preset rounds, then Gk=G_new, S4 would be executed, and a next round of testing would be started; otherwise, the test of the test template test_case_template would be ended.
The vectors in the set G_new satisfied the following conditions:
(1) In order to make the new test cases approximate to the boundary as much as possible, the closer the Rk boundary was, the denser the vector distribution was.
(2) In order to reduce the repeated testing of similar cases, there was a certain distance between the newly sampled vector and the tested vector, and the following conditions were met:
The absolute projection value of g−g_old in the jth dimension was greater than or equal to djk, where g_old is an arbitrary vector belonging to the union of G1, G2, . . . , Gk, and djk is the threshold of the distance between g and g_old in the jth dimension in the kth round.
As the number of iteration rounds increased, the accumulated vectors gradually increased. In order to reduce the amount of data calculation, djk was set to be a gradually decreasing value, namely:
d
jk=alpha*djk−1
where alpha is a given normal number, 0<alpha<1.
In an embodiment of automatic driving, alpha is 0.9.
The intelligent test method of the present disclosure is applied to the automatic driving scene, which can influence the algorithm logic of automatic driving case selection, so that the case that is more likely to be tested as the junction of low and normal is selected, and the training efficiency of automatic driving in the virtual training ground is increased.
It can be understood by those skilled in the art that the above examples are only preferred examples of the present disclosure, and are not intended to limit the present disclosure. Although the present disclosure has been described in detail with reference to the above examples, it is still possible for those skilled in the art to modify the technical solutions described in the above examples or equivalently replace some of the technical features. Modifications, equivalents and substitutions made within the spirit and principle of the present disclosure shall be included in the scope of protection of the present disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202111405653.4 | Nov 2021 | CN | national |
