Patent Application
20240320135
Continuous integration of software involves integrating working copies of software into mainline software, in some cases several times a day. Before integrating the working copy of software, the working copy must be tested to ensure it operates as intended. Testing working copies of software can be time consuming, especially when following typical testing protocols which require executing an entire test plan every test cycle. An entire test plan often takes hours to complete, which wastes computing resources and developer time. What is needed is an improved method for testing working copies of software.
The present technology, roughly described, automatically reduces the time to execute software testing through intelligent test selection and execution. The present system automatically detects what tests to execute based on code that has been changed, which is a subset of the entire list of tests to run for the block of code. Once the subset of tests is identified, annotations for tests are processed to update the subset as desired by the code administrator. Once updated, the system then automatically obtains the tests for the updated subset of tests. The tests to be executed are then distributed into groups or buckets. The distribution is set so that each group of tests will have as close to the same execution time as possible. The tests in each group or bucket are then executed concurrently with the other grouped tests. By running tests within the buckets concurrently, and with the total test execution duration as close to the same time as possible, the total testing duration is reduced to make the test execution as efficient as possible.
In some instances, the present technology provides a method for automatically testing software code concurrently. The method begins by detecting a test event initiated by a testing program and associated with testing a first software at a testing server. The test event is detected by an agent executing within the testing program at the testing server, and the testing event is associated with a plurality of tests for the first software. The method continues by receiving, by the agent on the testing server from the remote server, a list of tests to be performed in response to the test event. The received list of tests is a subset of the plurality of tests. The method then divides the subset of tests into two or more groups. Next, the method executes each of the two or more groups of tests concurrently.
In some instances, a non-transitory computer readable storage medium includes embodied thereon a program, the program being executable by a processor to automatically testing software code concurrently. The method begins by detecting a test event initiated by a testing program and associated with testing a first software at a testing server. The test event is detected by an agent executing within the testing program at the testing server, and the testing event is associated with a plurality of tests for the first software. The method continues by receiving, by the agent on the testing server from the remote server, a list of tests to be performed in response to the test event. The received list of tests is a subset of the plurality of tests. The method then divides the subset of tests into two or more groups. Next, the method executes each of the two or more groups of tests concurrently.
In some instances, a system for automatically testing software code concurrently includes a server having a memory and a processor. One or more modules can be stored in the memory and executed by the processor to detect a test event initiated by a testing program and associated with testing a first software at a testing server, the test event detected by an agent executing within the testing program at the testing server, the testing event associated with a plurality of tests for the first software, receive, by the agent on the testing server from the remote server, a list of tests to be performed in response to the test event, the received list of tests being a subset of the plurality of tests, divide the subset of tests into two or more groups, and execute each of the two or more groups of tests concurrently.
The present technology, roughly described, automatically reduces the time to execute software testing through intelligent test selection and execution. The present system automatically detects what tests to execute based on code that has been changed, which is a subset of the entire list of tests to run for the block of code. Once the subset of tests is identified, annotations for tests are processed to update the subset as desired by the code administrator. Once updated, the system then automatically obtains the tests for the updated subset of tests. The tests to be executed are then distributed into groups or buckets. The distribution is set so that each group of tests will have as close to the same execution time as possible. The tests in each group or bucket are then executed concurrently with the other grouped tests. By running tests within the buckets concurrently, and with the total test execution duration as close to the same time as possible, the total testing duration is reduced to make the test execution as efficient as possible. As a result, the test time reduction directly translates to dollars saved due to lower infrastructure usage as well as improved developer productivity.
The present system addresses a technical problem of efficiently testing portions of software to be integrated into a main software system used by customers. Currently, when a portion of software is to be integrated into a main software system, a test plan is executed to test the entire test portion. The entire test plan includes many tests and takes a long time to complete, often hours, and takes up large amounts of processing and memory resources, as well as time.
The present system provides a technical solution to the technical problem of efficiently testing software by intelligently selecting a subset of tests from a test plan and executing the subset. The present system identifies portions of a system that have changed or for which a test has been changed or added, and adds the identified tests to a test list. An agent within the test environment then executes the identified tests. The portions of the system can be method classes, allowing for a very precise list of tests identified for execution. The testing is performed more efficient by dividing the testing code into groups or buckets and executing the test groups concurrently.
Network 140 may be implemented by one or more networks suitable for communication between electronic devices, including but not limited to a local area network, wide-area networks, private networks, public network, wired network, a wireless network, a Wi-Fi network, an intranet, the Internet, a cellular network, a plain old telephone service, and any combination of these networks.
Testing server 110 may include testing software 120. Testing software 120 tests software that is under development. The testing software can test the software under development in steps. For example, the testing software may test a first portion of the software using a first step 122, and so on with additional steps through an nth step 126.
A testing agent 124 may execute within or in communication with the testing software 120. The testing agent may control testing for a particular stage or type of testing for the software being developed. In some instances, the testing agent may detect the start of the particular testing, and initiate a process to identify which tests of a test plan to execute in place of every test in the test plan. Testing agent 124 is discussed in more detail with respect to
Intelligence server 150 may communicate with testing server 110 and data store 160, and may access a call graph stored in data store 160. Intelligence server 150 may identify a subgroup of tests for testing agent 124 to execute, providing for a more efficient testing experience at testing server 110. Intelligence server 150 may, in some instances, apply annotations to a test subset, distribute tests into groups or buckets, move tests between groups or buckets, automatically obtain testing code, and perform other functionality as described herein. Intelligence server 150 is discussed in more detail with respect to
Data store 160 may store a call graph 162 and may process queries for the call graph. The queries main include storing a call graph, retrieving call graph, updating portions of a call graph, retrieving data within the call graph, and other queries.
The present application describes a system for testing software. Some additional details for modules described herein are described in U.S. patent application Ser. No. 17/371,127, filed on Jul. 21, 2021, titled “Test Cycle Time Reduction and Optimization,” and U.S. patent application Ser. No. 17/545,577, filed on Dec. 8, 2021, titled “Reducing Time to First Failure,” the disclosures of which are incorporated herein by reference.
Intelligence server 300 may include more or fewer modules than described with respect to
In some instances, the code to be tested is updated, or some other event occurs and is detected which triggers a test. A complete set of tests for the code may be executed at step 415. In some instances, a complete set of tests is run over time as updates to the set of tests eventually include every test, rather than executing every test at once at step 415. In some instances, all tests are not executed at step 415.
A call graph may be generated with relationships between methods and tests, and stored at step 420. Generating a call graph may include detecting properties for the methods in the code. Detecting the properties may include retrieving method class information by an intelligence server based on files associated with the updated code. The call graph may be generated by the intelligence server and stored with the method class information by the intelligence server. The call graph may be stored on the intelligence server, a data store, or both.
In some instances, generating the call graph begins when the code to be tested is accessed by an agent on the testing server. Method class information is retrieved by the agent. The method class information may be retrieved in the form of one or more files associated with changes made to the software under test. The method class information, for example the files for the changes made to the code, are then transmitted by the agent to an intelligence server. The method class information is received by an intelligence server from the testing agent. The method class information is then stored either locally or at a data store by the intelligence server.
A test server initiates tests at step 425. The agent may detect the start of a particular step in the test at step 430. A subset of tests is then selected for the updated code based on the call graph generated by the intelligence server at step 435. Selecting a subset of tests may include accessing files associated by the changed code, parsing the received files to identify method classes associated with those files, and generating a test list from the received method classes using a call graph. Selecting a subset of tests for an updated code based on the call graph is disclosed in U.S. patent application Ser. No. 17/371,127, filed Jul. 9, 2021, titled “Test Cycle time Reduction and Optimization,” the disclosure of which is incorporated herein by reference.
The test subset may be modified based on test annotation data at step 440. The test subset may be modified by adding tests or removing tests, based on the annotation data associated with one or more tests. The annotation data may be added to one or more tests in a variety of ways. In some instances, an administrator may add annotation data to one or more tests. The administrator may add annotation data (e.g., meta data, a label, or some other data) to indicate how a particular test should be handled. In some instances, a testing system may use logic to add annotation data to a particular test. More detail for modifying a test subset based on annotation data is discussed with respect to the method of
Test code is automatically obtained for each test to be performed in the modified test subset at step 445. The actual test code can be obtained in different ways, based on the system being tested and the platform. For example, in Java, the tests a particular changed portion of code may be in a test domain that can be accessed based on the name of the code that was modified. In some instances, the testing code can be obtained based on a blob specified by a code administrator.
A number of buckets are identified, by the present system, in which to execute subset tests at step 450. In some instances, the number of buckets may be set based one or more factors, including but not limited to a customer plan (paid, premium, not paid), number of resources available, number of tests to be executed in the subset, the capacity of each bucket, and so forth.
The present description uses the terms bucket and group to describe where tests are distributed. In some instances, the tests are grouped into buckets. However, the terms are intended to be interchangeable, and a bucket and group are not intended to be exclusive of each other.
The test duration for each test within a bucket is identified at step 455. For tests that have been executed previously, the test duration is determined as the average time of the previous test executions for that particular test. For tests that have not been executed previously, the test duration is determined based on the number of lines of code for that particular test. To determine the time to allocate for each line of code, the average execution time per line of code for other portions of code could be determined, and then applied to the lines of code for the test that has not been executed. The time per line of code could also be assigned by an administrator.
Subset tests are distributed into the identified buckets at step 460. The tests may be distributed such that each bucket has a total test execution time as close as possible to the average test execution time for all the buckets. This allows for the maximum time savings benefit during test execution, as no bucket should take much longer to execute than any other bucket. Distributing the subset tests into the identified buckets to maximize time savings is discussed in more detail below with respect to the method of
Once the subset of tests are within their respective buckets, a test agent may execute the tests in each bucket concurrently at step 465. For each bucket, the tests are run consecutively, and the first test in each bucket can be started simultaneously. In some instances, a test agent executes the test list with instrumentation on. This allows data to be collected during the tests. As the tests execute, data regarding each test is stored. The stored data includes whether the test passes or fails, the total execution time, whether the entire test executed, and other data.
Once the test execution is complete, an agent parses the test results and uploads the results with a newly automatically generated call-graph at step 470. The new call-graph is generated based on the results of the newly executed subset of tests.
At test completion, the testing agent accesses and parses the test results and uploads the results with an automatically generated call graph at step 455. Parsing the test results may include looking for new methods as well as results of previous tests. The results may be uploaded to the intelligence server and include all or a new portion of a call graph or new information from which the intelligence server may generate a call graph. The intelligence server may then take the automatically generated call graph portion and place it within the appropriate position within a master call graph. The call graph is then updated, whether it is stored locally at the intelligence server or remotely on the data store.
Test annotated with “Skip” are removed from the test subset at step 515. These tests should not be in the subset regardless of whether or not they were selected to be included within the subset of tests. The subset can further be updated based on other annotation data at step 520. For example, a particular test may be included based on conditions, such as whether one or more other tests are included or are not included. Some tests may be included based on the time, day, or total duration of the current subset of tests. Some tests may be included based on the platform for which the tests are being run.
Once the tests have been placed into buckets, the group tests may be adjusted to make each bucket of tests have a total test execution duration as close to the overall average bucket test execution duration as possible at step 620. Adjusting the group of tests within each bucket may include moving one or more tests from one bucket to another. Adjusting group tests within the buckets is discussed in more detail below with respect to the method of
Next, a first bucket is selected at step 415. A determination is then made as to whether the selected bucket has an execution time longer than the average bucket execution time at step 420. If the selected bucket does not have a longer execution duration than the average bucket execution duration determined at step 410, the method of
The selected bucket test is moved to a bucket that is below the average by an amount closest of the selected bucket test duration at step 430. Continuing the example, if a test is selected at step 425 that is 30 seconds long, it would be placed in another bucket having a duration underage, below the average duration, that is closest to 30 seconds. The method then continues to step 435.
A determination is made as to whether more buckets have an execution duration over average execution duration at step 435. If there are additional buckets having an execution duration over the average execution duration, the next bucket is selected at step 440 and the method continues to step 420.
If there are no additional buckets having an execution duration greater than the average duration, the method ends at step 745.
The third column in the table of
The table of
In some instances, the splitting/dividing of the tests into groups is based on the execution time data from the tests, and the splitting is an adaptive process. A split of the subset of tests will change over time, for example based on the previous execution data, because tests evolve with time.
The components shown in
Mass storage device 1130, which may be implemented with a magnetic disk drive, an optical disk drive, a flash drive, or other device, is a non-volatile storage device for storing data and instructions for use by processor unit 1110. Mass storage device 1130 can store the system software for implementing embodiments of the present invention for purposes of loading that software into main memory 1120.
Portable storage device 1140 operates in conjunction with a portable non-volatile storage medium, such as a floppy disk, compact disk or Digital video disc, USB drive, memory card or stick, or other portable or removable memory, to input and output data and code to and from the computer system 1100 of
Input devices 1160 provide a portion of a user interface. Input devices 1160 may include an alpha-numeric keypad, such as a keyboard, for inputting alpha-numeric and other information, a pointing device such as a mouse, a trackball, stylus, cursor direction keys, microphone, touch-screen, accelerometer, and other input devices. Additionally, the system 1100 as shown in
Display system 1170 may include a liquid crystal display (LCD) or other suitable display device. Display system 1170 receives textual and graphical information and processes the information for output to the display device. Display system 1170 may also receive input as a touch-screen.
Peripherals 1180 may include any type of computer support device to add additional functionality to the computer system. For example, peripheral device(s) 1180 may include a modem or a router, printer, and other device.
The system of 1100 may also include, in some implementations, antennas, radio transmitters and radio receivers 1190. The antennas and radios may be implemented in devices such as smart phones, tablets, and other devices that may communicate wirelessly. The one or more antennas may operate at one or more radio frequencies suitable to send and receive data over cellular networks, Wi-Fi networks, commercial device networks such as a Bluetooth device, and other radio frequency networks. The devices may include one or more radio transmitters and receivers for processing signals sent and received using the antennas.
The components contained in the computer system 1100 of
The foregoing detailed description of the technology herein has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen to best explain the principles of the technology and its practical application to thereby enable others skilled in the art to best utilize the technology in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the technology be defined by the claims appended hereto.
