Patent Application
20190327160
The present invention relates generally to a system and method for scheduling and executing. More specifically, the present invention relates generally to a system and method for scheduling and executing automated tests.
Currently there are many challenges in triaging and defect management of automated tests. Although the test execution itself is automated, the triaging and defect management process is manual. This analysis requires Software Development Engineer in Test (SDET's), domain teams, and functional subject matter experts (SME's) to debug and identify failure causes which takes time consuming effort.
Accordingly, there exists a need for automated tests that may be executed at the right time, modified, and re-run to provide a successful outcome. The development of the system and method for scheduling and executing automated tests fulfills this need.
It is thus a desired object of the present invention to provide a system for test failure analysis and improvement, comprising a processor system which is responsible for fetching, decoding, executing and writing back a plurality of test failure analysis and improvement data. The system also comprises a plurality of databases and a test failure analysis and improvement non-transitory storage media. The test failure analysis and improvement non-transitory storage media resides on the databases with the test failure analysis and improvement data residing on the test failure analysis and improvement non-transitory storage media. The databases are in electronic communication with the processor system.
The system also provides for a plurality of server logs, a plurality of test result logs having an integrated dashboard; an application performance management tool which links test executions with the server logs and the test result logs each having a test session ID. The test failure analysis and improvement non-transitory storage media generates a special log token for each of the test session IDs. The system also provides a platform agnostic test runner integrating with the application performance management tool to link test executions with the server logs and the test result logs through an application performance management service invoker which abstracts all tool integration details and speeds-up a failure analysis process.
The system also provides a test execution log which contains the application performance management tool information for each test execution. The test execution log speeds-up the failure analysis process. Additionally, the system also provides an automation test execution and triaging pipeline which receives guidance from a continuous integration/continuous delivery orchestrator that automates the test execution and a triage process and a failure analytics engine building a root cause analysis database. The root cause analysis database is a dynamic knowledge database which increases its accuracy of its RCA and resolution.
The system also comprises a circuit breaker which functions as a test controller component. The circuit breaker is a platform agnostic component that ensures a high success rate for executing tests by executing tests. The server logs, the test result logs, the integrated dashboard, the application performance management tool, the test session IDs, the special log tokens, the platform agnostic test runner, the APM service invoker, the platform agnostic component, the test execution log, the automation test execution and triaging pipeline, the continuous integration/continuous delivery orchestrator, the failure analytics engine, and the circuit breaker reside on the performing test data management non-transitory storage media.
The overall system automates the triaging process for test failures thereby ensuring high quality defect resolutions, reduced defects and reduced time for the triaging process. The overall system also includes a platform agnostic product that greatly enables and speeds up the triaging process and enables successful execution of end to end tests in shared unstable quality analysis test environments. The overall system calculates a high success rate probability time for each test and executes and schedules each of the tests for execution during that time enabling a high number of successful test executions in shared QA test environments.
The databases may include a root cause analysis database and an environment downtime tracker database. The integrated dashboard may speed-up the test failure analysis and ensure that one or more proper RCAs are provided for issue resolution. The test execution log may include a pass/fail status, one or more failure error traces and a plurality of linked server and application stats. The root cause analysis database may be based on a plurality of test failures, a plurality of server logs, and a root cause analysis provided by the triage team.
The circuit breaker may calculate the high success rate based on environment health status, application health status, previous similar test runs, and application or environment downtime. The system may further comprise failed/stopped tests that cannot be run due to an environment issue will be auto-scheduled for re-run using a test scheduler. The circuit breaker may also integrate with a testing tool selected from the group consisting of Junit, TestNG, or CA Dev Test using a test execution controller. The test execution controller may result in a platform agnostic manner that is consumed by application performance management tool.
A method for test failure analysis and improvement, comprising the steps of firstly, obtaining a system for performing test data management; secondly, executing a plurality of tests for a plurality of test failures; thirdly, collecting a plurality of application and environment health stats; fourthly, testing an execution history and a plurality of application health information; fifthly analyzing testing results; and lastly performing a plurality of failed/stopped tests. The tests may be performed by a platform agnostic test runner. The collecting step may be performed by the application monitoring tool while the analyzing step may be performed by a circuit breaker. The testing step may be performed with root cause analysis failure.
A non-transitory computer storage media having instructions stored thereon is also provided which, when executed, execute a method comprising the steps of: firstly, obtaining a system for performing test data management; secondly, executing a plurality of tests for a plurality of test failures; thirdly, collecting a plurality of application and environment health stats; fourthly, testing an execution history and a plurality of application health information; fifthly, analyzing testing results; and lastly, performing a plurality of failed/stopped tests. The tests may be performed by a platform agnostic test runner while the collecting step may be performed by the application monitoring tool. The analyzing step may be performed by a circuit breaker while the testing step may be performed with root cause analysis failure.
The advantages and features of the present invention will become better understood with reference to the following more detailed description and claims taken in conjunction with the accompanying drawings, in which like elements are identified with like symbols, and in which:
10. System for test failure analysis and improvement
15. Server logs
20. Application performance management (APM) tool
25. Test result logs
30. Integrated dashboard
35. Test environment
40. Platform agnostic test runner
45. APM service invoker
50. Test execution log
55. Triage team
60. IT support team
65. Automation test execution and triaging pipeline
70. Continuous integration/continuous delivery (CI/CD) orchestrator
75. Failure analytics engine
80. Root cause analysis (RCA) database
85. Circuit breaker
90. Failed/stopped tests
95. Test scheduler
100. Test execution controller
The best mode for carrying out the invention is presented in terms of its preferred embodiment, herein depicted within
The terms “a” and “an” herein do not denote a limitation of quantity, but rather denote the presence of at least one (1) of the referenced items.
Referring now to
The system 10 operates by linking tests executions with application logs, service logs, and environment logs through integration with APM tools using “test session ID's”. These test sessions ID's are special log tokens that are generated by the system 10 which are then consumed by tests, APM tools, applications and get logged enabling the system 10 to link related logs together.
The system 10 integrates with APM tools 20 which happens in a platform agnostic test runner 40 through an APM service invoker 45 which abstracts all tool integration details. This process then publishes a test execution log 50 containing information including, but not limited to: pass/fail status, failure error traces and linked server and application stats to the application performance management (APM) tool 20 for each test execution. This process greatly speeds up the failure analysis process as a triage team 55 can consider the integrated dashboard 30 and identify the problem areas very fast and route to an IT support team 60 for resolution. This process also greatly speeds up the resolution process as all debug information such as logs, request/response, and test data available in single view for analysis.
An automation test execution and triaging pipeline 65, receiving guidance from a continuous integration/continuous delivery (CI/CD) orchestrator 70 automates the test execution and triaging process. The pipeline component is template based enabling customization of the base template to meet organization's test execution and triaging process. A failure analytics engine 75 will build a root cause analysis (RCA) database 80 based on test failures, server logs, RCA provided by triage team etc. The root cause analysis (RCA) database 80 is not a static database but a dynamic knowledge base which increases its accuracy of its RCA and resolution using learning algorithms and analytics. The data, published to the integrated dashboard 30 raises defects for each test failures, tracks each defect to proper resolution and closing of the defect. Additionally, it routes defects to the IT support team 60 that provide resolution of the defects and ensures that proper RCAs are provided for issue resolution. As such, it greatly reduces the time taken to resolve test failures from days to hours/minutes, it reduces the number of poor-quality defects, and increases quality in overall defect management and resolution process.
A circuit breaker 85, functioning as a test controller component, ensures a high success rate for executing tests by executing tests in a right time window where it will have a high probability of running successfully. This circuit breaker 85 calculates the “high success rate” probability using decision-based algorithms based on factors such as environment health status, application health status, previous similar test runs, and application or environment downtime. Failed/stopped tests 90 that cannot be run due to environment issues will be auto-scheduled for re-run using a test scheduler 95. The circuit breaker 85 is a platform agnostic component that can integrate with different industry testing tool such as Junit, TestNG, CA Dev Test etc. using a test execution controller 100. The test execution controller 100 provides the logic required to invoke tests on different testing tools. It also logs test execution logs and results in platform agnostic manner that can be consumed by application performance management (APM) tool 20. As such the test execution controller 100 provides multiple benefits including but not limited to complete automated test executions with no manual intervention and ensures high probability of test success in shared QA environments by using decision-based algorithms to check if test execution will be successful.
The preferred embodiment of the present invention may be utilized by the common user in a simple and effortless manner with little or no training. It is envisioned that the system 10 would be constructed in general accordance with
The system 10 brings efficiency and speed by reducing laborious manual testing tasks typically performed by triage team 55 and IT support team 60 as well as completing overall testing quickly. A circuit breaker 85 applies design patterning to any automated testing strategy or framework for continuous monitoring of current state of the application environment. The circuit breaker 85 stops or breaks the test execution when the environment becomes unstable, and then re-executes the test scripts when the environment is once again stable. The present invention will run in the background as heart beat monitor, validate the current, complete, good and continuous availability of environment, launch appropriate test script that need to be executed, break any potentially failing tests due to such instability and rerun the scripts back when the environment is stable.
A method for test failure analysis and improvement includes the steps of obtaining a system for performing test data management, executing a plurality of tests for a plurality of test failures, collecting a plurality of application and environment health stats, testing an execution history and a plurality of application health information, analyzing testing results and performing a plurality of failed/stopped tests.
The obtaining step includes logging-into the system for performing test data management. The executing step includes the tests are performed by a platform agnostic test runner. The collecting step is performed by the application monitoring tool. The testing step is performed with root cause analysis failure. The analyzing step is performed by a circuit breaker. The performing step is performed by a test scheduler.
The method is performed by a test failure analysis and improvement non-transitory computer storage media having instructions stored thereon which, when executed, execute a method comprising the steps of obtaining a system for performing test data management, executing a plurality of tests for a plurality of test failures, collecting a plurality of application and environment health stats, testing an execution history and a plurality of application health information, analyzing testing results and performing a plurality of failed/stopped tests.
The obtaining step includes logging-into the system for performing test data management. The executing step includes the tests are performed by a platform agnostic test runner. The collecting step is performed by the application monitoring tool. The testing step is performed with root cause analysis failure. The analyzing step is performed by a circuit breaker. The performing step is performed by a test scheduler.
The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible considering the above teaching. The embodiments were chosen and described to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the use contemplated.
The present application claims the benefit of U.S. Provisional Patent Application 62/659,165 filed on Apr. 18, 2018, the entire disclosure of which is incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62659165 | Apr 2018 | US |
