System and methods for automated detection of input and output validation and resource management vulnerability

Abstract

In an example embodiment, a system analyzes a set of computer routines. The system may perform an analysis including a determination of a likelihood of vulnerability to unexpected behavior for one or more computer routines of the set. Based upon the analysis, the system may identify one or more computer routines of the set having the likelihood of vulnerability. The system may asynchronously and dynamically manipulate at least one of the one or more computer routines through a testing technique. The system may determine unexpected behavior of at least one of the one or more computer routines.

Description

BACKGROUND

Testing even simple commercial computer applications can be extremely complex because the number of independent code paths to be tested can be very large. Each of these code paths, in turn, is composed of a very large number of individual functions, which may be composed of one or more blocks of non-adjoining instructions that further complicate testing. There is a need in the computer industry for an approach that performs such complex testing in an efficient manner.

SUMMARY

The present disclosure is directed to systems and methods that automate detection of input/output validation (e.g., testing) and output resource management vulnerability. The systems and methods may analyze a set of computer routines. The analysis may include a determination of a likelihood of vulnerability to unexpected behavior for one or more computer routines of the set. Based upon the analysis, the systems and methods may identify the one or more computer routines of the set having the likelihood of vulnerability. The systems and methods may asynchronously and dynamically manipulate at least one of the one or more computer routines through a testing technique. The systems and methods may determine unexpected behavior of at least one of the one or more computer routines.

In some embodiments, the systems and methods may deploy one or more patches to correct the unexpected behavior of at least one of the one or more computer routines. In some embodiments, the systems and methods may analyze the set of computer routines and at least one corresponding sequence of computer routines of the set.

In some embodiments of the systems and methods, the analysis may further include at least one of the following: extracting a histogram including a frequency of usage associated with at least one computer routine of the set, determining size of one or more buffer (e.g., memory segment) read or write computer operations associated with the one or more computer routines, determining the size of one or more corresponding stacks associated with the one or more computer routines, determining size of one or more memory read or write operations based upon examining a corresponding loop size, and performing taint analysis of at least one computer routine of the set. The histogram may include, but is not limited to, at least one of the following: a log file, a graph, a table, other user display and other types of display. Some embodiments may include one or more computer threads. Some embodiments may include two or more computer threads (e.g., multi-threaded). In some embodiments, a computer thread (e.g., computer thread of execution) may represent the smallest sequence of programmed instructions that can be managed independently by a scheduler (e.g., a method by which resources are assigned to complete work), which may be part of the computer operating system. In some embodiments, the computer threads may include a sequence of computer routines which may include at least (e.g., one or more) of function calls and system calls. According to some embodiments, the histogram may depict how many times a given function or system call of a computer thread of a computer application is executed.

In some embodiments of the systems and methods, the one or more computer routines may include at least one (or more) of: a function and a system call. Some embodiments of the systems and methods may manipulate the at least one of the one or more computer routines by at least one of the following: modifying data associated with the one or more computer routines, the data exceeding a corresponding buffer (e.g., memory segment) size, and modifying values that are declared in memory regions associated with (e.g., accessed by) the one or more computer routines.

Some embodiments of the systems and methods may determine unexpected behavior of at least one of the one or more computer routines including determining that a control flow of a thread associated with the one or more computer routines has changed as a result of the manipulation, determining a failure condition that caused the thread to change its control flow, and displaying the failure condition.

In some embodiments of the systems and methods, for at least one function of the one or more computer routines, the computer testing technique may provide at least one of invalid, unexpected, and random data to at least one of an input of the at least one function, logic within the at least one function, and an output of the at least one function. In some embodiments of the systems and methods, for at least one system call of the one or more computer routines, the computer testing technique may provide at least one of invalid, unexpected, and random data to a system call parameter associated with the at least one system call.

In some embodiments of the systems and methods, the system call parameter may be associated with at least one of the following: thread synchronization, process synchronization, thread scheduling, process scheduling, memory, memory allocation, memory de-allocation, memory writing, memory reading, a network socket, creation of a network socket, network socket input, network socket output, pipe creation, system input, system output, shared memory fifo creation, a terminal input, a terminal output, file handling, file creation, file writing, file reading, disk input, and disk output.

In some embodiments, the systems may include an analysis engine. The systems may also include a validation engine that may be communicatively coupled to the analysis engine (e.g., threads and processes being examined). The systems may also include an instrumentation engine that may be communicatively coupled to at least one of the analysis engine and the validation engine.

In some embodiments, the analysis engine and the validation engine may comprise a processor fabric including one or more processors. In some embodiments, the analysis engine, the validation engine, and an instrumentation engine may comprise a processor fabric including one or more processors.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particular description of example embodiments of the disclosure, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present disclosure.

FIG. 1A illustrates a given function that may include functionality and also may include error handling capabilities, in embodiments of the present disclosure.

FIG. 1B illustrates the state of a given function, according to embodiments of the present disclosure.

FIG. 2 illustrates execution of a computer application creating one or more threads that, in turn, may execute one or more functions, in embodiments of the present disclosure.

FIG. 3 illustrates a flowchart of an example method (and system) for automating detection of input/output validation and output resource management vulnerability, in embodiments of the present disclosure.

FIG. 4A illustrates how one or more features in a computer application may include one or more independent code paths, which in turn may execute one or more different sets of functions in embodiments of the present disclosure.

FIG. 4B illustrates an example operating environment (e.g., application environment) of the computer application, according to embodiments of the present disclosure.

FIG. 4C illustrates different types of system resources, according to embodiments of the present disclosure.

FIG. 5A illustrates a histogram according to embodiments of the present disclosure.

FIG. 5B illustrates a list of system resources on which stress testing may be performed and a list of attack vectors, according to embodiments of the present disclosure.

FIG. 6 illustrates the results of a stress test on a dashboard, according to embodiments of the present disclosure.

FIG. 7A illustrates an example block diagram of the client and analysis engine in embodiments of the present disclosure.

FIG. 7B illustrates an example protocol data unit (PDU) used to transmit data between the client and an analysis engine of FIG. 7A.

FIG. 8 illustrates a computer network or similar digital processing environment in which embodiments of the present disclosure may be implemented.

FIG. 9 illustrates a diagram of an example internal structure of a computer (e.g., client processor/device or server computers) in the computer system of FIG. 8.

DETAILED DESCRIPTION

A description of example embodiments of the disclosure follows.

The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.

Some embodiments may help in improving robustness of not only the basic functionality of software (e.g., computer applications), but also software error handling functionality, and may also exercise those code paths that are too difficult to reach when a test suite designed to exercise and test the functionality of the application is executed. In some embodiments, such software testing can be automated in a user friendly manner. In some embodiments, the status of software testing may be displayed to testers, developers, and management of the company developing the software under test.

Multi-State Stress and Computer Application Execution:

FIG. 1A is an illustration of an embodiment 100 that includes a given function 106 that may include functionality 102 and also may include error handling capabilities 104, according to embodiments of the present disclosure. Individual functions 106 may include computer code (also known as “code”) for basic functionality 102 as well as code for error handling scenarios 104. Computer applications may be constructed using a plethora of internal, but independent, code paths that, collectively, may implement functionality offered by a given computer application. A function may include code that implements basic functionality 102, or code that implements error handling 104 capabilities, or a combination of both types of code. Some computer code may be intended for basic functionality 102 (e.g., known as “sunny side” code, or basic functionality code) and not specifically directed to handle errors. By contrast, error handling code 104 may implement functionality to recover when the basic functionality code encounters unexpected run time situations, including but not limited to lack of a resource and/or unexpectedly large input data.

Testing of computer routines (including but not limited to functions, system calls, and other types of computer routines) to detect and handle errors may be complex. Cyclomatic complexity is a measure of the number of linearly independent paths (e.g., independent code paths) through a computer application's source code. Some computer application may be very complex and more difficult to stress test. As such, error handling in larger computer applications is of great importance. Therefore, some embodiments exercise and check the error handling functionality (e.g., capabilities) of computer routines (including but not limited to functions or system calls).

FIG. 1B illustrates the state of a computer routine (including, but not limited to functions, API calls and system calls), according to embodiments of the present disclosure. As illustrated in FIG. 1B, in some embodiments, a computer routine 170 (including but not limited to a function and system call) on each independent code path may be subjected to stress (e.g., stress tests or manipulation of the computer routine) at one or more of the following three states: (1) at the start 172 of a computer routine on an independent code path, (2) in body 174 (or middle) of a computer routine on an independent code path, and, (3) at the end 176 of execution of a computer routine on an independent code path.

In some embodiments, differences exist between the states. In some example embodiments, (1) applying stress on an input may help to find the function or system call's vulnerability against erroneous (e.g., bad) input. In some example embodiments, (2) changing data in the body of the function may serve to exercise code paths in the function that are not otherwise easily exercised. In some example embodiments, (3) artificially changing the output values may create unusual conditions, so that error conditions and/or exception handling code may be exercised. According to some example embodiments, the first two states (1) and (2) may therefore be considered as “Code Exercise Test” states and the third state (3) may be considered as a “Negative Test” state.

However, in some embodiments, such stress (e.g., stress tests) are not limited to being applied at only three states, and may be applied at four or more states. Some embodiments may include a fourth state (4) which may be time dependent. In some example embodiments, for code that executes repeatedly, stress may be applied to a given instance of invocation. In an example embodiment, stress may be applied on the N-th (e.g., first, hundredth, or other number) instance of execution.

Some embodiments, before applying stress testing, may identify at least one of critical and high value computer routines (e.g., at least one of functions, system calls, and other computer routines). As such, to follow, this disclosure describes methods (and systems) of identifying at least one of critical and high value functions and then subjecting these functions to the three-state stress described above.

FIG. 2 illustrates a computer application executing 200, which may create one or more threads (collectively, 210, 220, 230) that, in turn, may execute one or more functions (collectively, 212, 214, 216, 218, 222, 224, 226, 228, 232, 234, 236, and 238) in embodiments of the present disclosure. According to some embodiments, FIG. 2 may be considered to represent the set of function call stacks for the threads (collectively 210, 220, 230). As mentioned above, the computer application under test may include several independent code paths. Some embodiments may overcome deficiencies of existing approaches, such as deficiencies in which it is difficult to test a number of desired code paths in a timely manner. Given that software product vendors may make a commercial decision to release a product when a certain percentage of independent code paths are tested, it is preferable to generate a prioritized list of functionality that is critical for the functioning of the application. Some embodiments test at least these critical functionality, enabling software vendors to lower the risk of software failing. In some embodiments, remaining independent code paths can also be tested and product patches may be released. Some embodiments may execute a test suite, and then may extract a list of critical functionality by determining which functions may be executed more frequently than other functions (or which functions take a greater percentage of time when the test suite runs). The list of critical functionality may include a set of call graphs for one or more threads within the computer application. The call graphs may include one or more continuous real time listings of functions from owned code, as well as from third party libraries, such as runtime and third party libraries used by a computer application. The call graph may further include one or more continuous real time listings of system calls interspersed between the aforementioned function calls made periodically by the computer application. In this manner, according to some embodiments, a complete real time state transition listing of the computing environment of each thread of the computer application may be captured.

Detection Process

FIG. 3 illustrates a flowchart of an example method for automating detection of input/output validation and output resource management vulnerability, in embodiments of the present disclosure. The flowchart in FIG. 3 illustrates a method (and system) 300 that automates detection of input/output validation and output resource management vulnerability. The method (and system) 300 may analyze a set of computer routines 302. The computer routines may include at least one of functions, API calls, system calls, and other types of computer routines. The analysis may include a determination of a likelihood of vulnerability to unexpected behavior for one or more computer routines of the set. Based upon the analysis, the method (and system) 300 may identify 304 the one or more computer routines of the set having the likelihood of vulnerability. For example, some embodiments may identify one or more computer routines, including but not limited to identifying functions and system call sequences used by one or more threads, for each use case, in order to map the run time control flow of a computer application. The method (and system) 300 may asynchronously (e.g., multi-threaded) manipulate at least one of the one or more computer routines through a testing technique 306. The method (and system) 300 may also dynamically (e.g., at run time, rather than at compile time) manipulate at least one of the one or more computer routines through a testing technique 306. However, the method (and system) is not so limited, and some embodiments may perform manipulation synchronously (e.g., single-threaded) and/or statically (e.g., at compile time, rather than run time).

As part of the manipulation, some example embodiments may fuzz (e.g., perform fuzz testing on, including but not limited to providing invalid, unexpected, and random data to the inputs of) the computer routine (e.g., including but not limited to a function or system call) with data larger than the buffer size (e.g., memory segment size) to examine if it is vulnerable to buffer error vulnerability. Some example embodiments may perform fuzzing using string inputs larger than the stack size.

Some example embodiments may fuzz numbers (e.g., which may include parameters, input and output values) that are declared in memory regions that may (or may not) be provided by users by one or more of the following: (1) some example embodiments may change numbers to be larger than a given architecture size (such as 8/32/64-bit or N-bit architecture size) if the underlying instructions are math operations, including but not limited to add, subtract, multiply and/or divide operations; (2) some example embodiments may change the sign of such numbers; (3) some example embodiments may change the value of such numbers to zero, if the disassembly (conversion of a program from its executable form into a form of assembler language that is readable by a human), shows division type operation and/or if the number is used as an address; and (4) other method(s) of fuzzing numbers. Numbers may include integers and/or floating point numbers (including but not limited to single-precision, double-precision, N-bit precision, and/or other types of precision) and may include a corresponding sign.

In order to achieve the manipulation of the one or more computer routines (for example, fuzz input and/or output values), some embodiments may modify one or more stacks (e.g., computer stack arrays) in computer memory. In order to achieve the manipulation, some embodiments may modify the stack pointer and/or the values within the stack. In order to achieve the manipulation, some embodiments may modify one or more of the following computer registers: the EAX (accumulator register), EBX (base register), ECX (counter register), EDX (data register), ESI (source index register), EDI (destination register), EBP (base pointer), and/or ESP (stack pointer), other registers, and other pointers.

The method (and/or system) 300 may determine unexpected behavior of at least one of the one or more computer routines 308. Some example embodiments may check to see if the control flow of the thread changed as a result of fuzzing by comparing the control flow extracted with and without that function or system call being attacked. Some example embodiments may identify the precise failure that caused the thread to change its control flow. Some embodiments may report the unexpected behavior to a display (e.g., dashboard) in the form of a failure condition being displayed, in a standard format, including but not limited to a syslog format and/or other formats.

In some embodiments the technique used to fuzz a function may include providing fake (e.g., false or unexpected) input and letting the function execute with the manipulated input. In some embodiments, a system call may execute normally, but fuzzing may overwrite the system call's result before it is read by an entity making the system call. As such, in some embodiments, the method (and system) of fuzzing may be different between functions and system calls.

In some embodiments, the method (and/or system) 300 may optionally deploy 310 one or more patches to correct the unexpected behavior of at least one of the one or more computer routines. In some embodiments, the method (and/or system) may analyze the set of computer routines and at least one corresponding sequence of computer routines of the set.

In some embodiments of the method (and/or system), the analysis 302 may further include at least one of the following. Some embodiments may extract a histogram including a frequency of usage associated with at least one computer routine of the set. Some example embodiments may extract a histogram of the most commonly used functions and system calls. Some embodiments may determine size of one or more buffer read or write computer operations associated with the one or more computer routines. Some embodiments may determine size of one or more corresponding stacks associated with the one or more computer routines. Some example embodiments may identify functions that include large buffer read and/or write operations and/or their corresponding stack sizes at the time of creation of the stack. Some embodiments may determine size of one or more memory read or write operations based upon examining a corresponding loop size. Some embodiments may perform taint analysis of at least one computer routine of the set. Some embodiments may identify at least one instruction associated with the one or more computer routines, the at least one instruction performing a computer operation that includes at least one of incrementing a value, decrementing a value, adding a value, subtracting a value, multiplying a value, and dividing a value. Some example embodiments may identify computer instructions (e.g., computer routines) that perform math operations, including but not limited to increment, decrement, add, subtract, multiply, and/or divide, of two or more numbers. Some example embodiments may determine if at least one of the two or more numbers are in user-provided input by performing taint analysis.

In some embodiments of the method (and/or system), the one or more computer routines may include at least one of: a function and a system call. Some example embodiments may identify the functions and system call sequences used by each thread for each use case to map the run time control flow of an application.

Some embodiments of the method (and/or system) may manipulate 306 the at least one of the one or more computer routines by at least one of the following: modifying data associated with the one or more computer routines, the data exceeding a corresponding buffer size, and modifying values that are declared in memory regions associated with the one or more computer routines.

Some embodiments of the method (and/or system) may determine unexpected behavior 308 of at least one of the one or more computer routines including determining that a control flow of a thread associated with the one or more computer routines has changed as a result of the manipulation, determining a failure condition that caused the thread to change its control flow, and displaying the failure condition.

In some embodiments of the method (and/or system), for at least one function of the one or more computer routines, the computer testing technique in 306 may provide at least one of invalid, unexpected, and random data to at least one of an input of the at least one function, logic within the at least one function, and an output of the at least one function. In some embodiments of the method (and/or system), for at least one system call of the one or more computer routines, the computer testing technique may provide at least one of invalid, unexpected, and random data to a system call parameter associated with the at least one system call.

In some embodiments, a system call parameter may include a return value of a system call. As such, in some embodiments, the system call return value may be “overwritten” by a known system call parameter (e.g., system call return value or system call error code) with a fake (false or unexpected) result.

In some embodiments of the system (and/or system), the system call parameter may be associated with at least one of the following: thread synchronization, process synchronization, thread scheduling, process scheduling, memory, memory allocation, memory de-allocation, memory writing, memory reading, a network socket, creation of a network socket, network socket input, network socket output, pipe creation, system input, system output, shared memory fifo creation, a terminal input, a terminal output, file handling, file creation, file writing, file reading, disk input, and disk output.

In some embodiments, the systems may include an analysis engine. The systems may also include a validation engine that may be communicatively coupled to the analysis engine (e.g., threads and processes being examined). The systems may also include an instrumentation engine that may be communicatively coupled to at least one of the analysis engine and the validation engine.

Computer Operating Environments and Resources:

FIG. 4A illustrates how one or more features 410 in a computer application may include one or more independent computer code paths (collectively, 402, 404, 406 in FIG. 4A), which, in turn, may execute one or more different sets of computer routines (collectively, 412, 414, 416, 420, 422, and 424 in FIG. 4A), which may include functions and/or system calls, in embodiments of the present disclosure. Although six computer routines (412, 414, 416, 420, 422, and 424) and three code paths (402, 404, and 406) are shown in FIG. 4A, embodiments are not so limited and may include several code paths (e.g., N or more code paths) and several computer routines (e.g, M or more computer routines).

In some embodiments, the computer routines (collectively, 412, 414, 416, 420, 422, and 424 in FIG. 4A) may execute in a certain sequence, in order to implement functionality associated with one or more use cases of a computer application. In some embodiments, the state of the computer application may be determined by examining the sequence of computer routines executed at a given moment of time.

In some embodiments, testing of software may be accomplished by a combination of manual and/or automated methods that exercise independent code paths. In some embodiments, the set of independent code paths may be a very large vector space. Therefore, the number of tests to be performed may be very large. Given that manual testing of the large vector space may be cumbersome, as timely creation of dependencies that may be required by the test suite rests squarely on the tester, some embodiments provide automated testing. Therefore, executing the test suite in a repeatable manner when feature-functionality changes may be challenging.

To overcome this challenge, some embodiments provide an advantage in that automated testing methods (and systems) of some embodiments do not require that dependencies including test files, input data, system resources, hardware to be ready prior to the launch of the test suite. As such, some embodiments overcome the deficiencies of existing approaches, which require dependencies to be available to avoid resulting failures.

FIG. 4B illustrates an example operating environment 430 of the computer application, according to embodiments of the present disclosure. As illustrated in FIG. 4B, in some embodiments, as the computer application executes 432, it may interact with other parts of the computer application 432, other libraries in the computer application 434, and/or the kernel (and/or operating system) 436.

As the test suite executes, the application may experience stress due to minor variations in the environment in which the application operates. In some embodiments, the operating environment 430 may include the kernel 436, the runtime libraries 434 and/or other libraries 434 used by the application as well as the external data that is presented to the application 432. To test the computer application 432, some embodiments create variations in the operating environment using automated mechanisms.

In some embodiments, other runtime stress may be introduced in real time through a runtime computer routine's (such as an API, function, or system call) “hooking” of kernel 436, library 434, and raw application 432 computer code. As known in the art of computer programming, “hooking” may include one or more of a range of techniques to alter behavior of applications, of an operating system, and/or other software components by intercepting computer routine calls, messages and/or events passed between software components (including but not limited to the software components illustrated in FIGS. 4A-4B).

In some embodiments, external data input stress may be introduced in real time by changing either the runtime arguments presented to a computer routine (including but not limited to an API and/or a function and/or system call), and/or by changing the body of a given computer routine at run time, and/or even be percolated upwards into the call stack by changing the return value of a given library and/or user functionality at run time.

FIG. 4C provides an illustration of different types of system resources (collectively 452, 454, 456, 458, 460, and 462) that may be associated with an operating system kernel 464, according to embodiments of the present disclosure. System Resource stress may be introduced via real time injection through the system call interface. Such real time injection may include manipulation of one or more inputs and/or outputs and/or behaviors of a system call, but is not so limited. System resource functionality 450 that may be so manipulated may include one or more of the following types of system resource functionality 450 (accessible through system calls and/or other means) but is not so limited: (1) thread and/or process synchronization 462; (2) thread and/or process scheduler 460; (3) memory, memory write/read, memory allocation/de-allocation 458; (4) network socket creation, network socket input and/or output 456; (5) terminal input and/or output 454; (6) disk input and/or output 452; and (7) pipe creation, system input/output, shared memory fifo creation, file handling, file creation, file writing, and/or file reading.

Stress-Testing Displays:

FIG. 5A illustrates a histogram 500 according to embodiments of the present disclosure. A histogram 500 may include a graphical representation of distribution of numerical data. The histogram may include, but is not limited to, a web dashboard, a log file, a graph, table, and/or other user display and/or other type of display. The histogram may enable a user to determine how many computer routine (e.g., API and/or function and/or system call) failures are detected. According to some embodiments, the histogram 500 may depict how many times 514 a given computer thread 510 (leftmost column of FIG. 5A, a thread having a sequence of computer routines which may include function calls and/or system calls) of a computer application as executed. The middle column of FIG. 5A illustrates transitions between computer routines (function and/or system calls) associated with the call graph for a given thread 510. In some embodiments, by identifying the most frequent threads of execution through a histogram metric 514 (such as time, and/or count which may include a number and/or numerical value, and/or another metric), a tester (e.g., automated test machine and/or process and/or end user tester) may prioritize a testing effort.

FIG. 5B illustrates a list of system resources on which stress testing may be performed for one or more threads 522, according to embodiments of the present disclosure. Each row of the table 520 in FIG. 5B may represent a specific attack vector. As in FIG. 5A, in FIG. 5B each thread 522 (collectively 522a, 522b, 522c) may have an associated transition 524 (collectively, 524a, 524b, 524c) and histogram metric 526 (collectively 526a, 526b, 526c). In FIG. 5B, the underlying stress may be applied to a stress type 532, including but not limited to a resource (e.g., system resource, or “resource”) and/or to an API (Application Programming Interface and/or function and/or “code exerciser”). The stress may be applied to a starting address 528 (collectively 528a, 528b, 528c) and/or ending address 530 (collectively 530a, 530b, 530c), which may be one or more addresses in computer memory (including but not limited to memory addresses and/or pointers on one or more computer stacks). The “new input” column 536 may represent one or more input values 536b, and/or code body values 536c, and/or return values 536a each of which may be applied in the stress test. In an example embodiment shown in FIG. 5B, a return value 536a of 0xDEADBEEF may be applied to a stress type 532 having a system resource 532a during a next action 534 associated with the function transition from function Fz to function Fr 534a. Also as illustrated in FIG. 5B, an input parameter 534b having input values 536b (0xAAAAAA and 0xBBBBBB) may be applied to a stress type 532 having a code exerciser 532b (including, but not limited to, a function exerciser). Also as illustrated FIG. 5B, replacement code 536b may be applied to a code body 534c as the next action in a code exerciser 532c stress test. In some embodiments, a tester may choose to apply stress in the input stage of the system call (and/or API and/or function call) 534b, and/or on the output stage of the system call 534a. As illustrated in FIG. 5B, in the case of system resource stress, a tester may change the return value 536a of the system call to make a higher layer of the computer application interpret that an underlying system call failed.

As such, FIG. 6 illustrates the pass/fail results 614 (614a, 614b, 614c) of a stress test on a computer dashboard 600 (also known as dashboard and/or web dashboard), according to embodiments of the present disclosure. In some embodiments, a computer dashboard may include a real-time, easy-to-read, and single page graphical presentation of status and trends. In some embodiments, a dashboard may be analogous to a report and/or progress report. In some embodiments, a dashboard may be continuously updated and/or periodically updated.

As illustrated in FIG. 6, each row may be considered as an attack vector, including a respective thread 602, transition 604, histogram metric 606, start address 608, end address 610, stress type 612, and pass or fail result 614. As illustrated in FIG. 6, by returning different failure codes, the tester may simulate various failure conditions and observe how the computer application responded. If the computer application thread 602 executes the next transition 604 as specified in the next action column, and/or if the computer application takes a handled exception, the computer application may be flagged as handling the real time stress appropriately. On the other hand, if the computer application takes an unhandled exception, the computer application may have an input validation and/or resource management vulnerability, in which case the computer application may be flagged as handling the real time stress inappropriately.

Once the list of targeted stress is set up, a test suite may be used to exercise the application under test. Some embodiments may cause the execution of the appropriate stress in the stress vector set up by the tester. For situations where the stress is a negative stress, which tests a code block's handling of invalid input and/or unexpected behavior in regards to the code block's functionality (e.g., exception handling), some embodiments may report the success and/or failure of the stress vector by observing the next transition after the test. Negative stress that causes the application to crash or hang may be reported on a dashboard and/or web portal 600. For code exerciser tests, normal testing may continue and other logical, performance, load, runtime, resource exhaustion, and security testing may continue.

In some embodiments, for each critical function transition on the call graph, a tester can set up the aforementioned variety of stress using a graphical user interface or the command line. In some embodiments, for each stress instance, a few fixed parameters may be provided. In some embodiments, these fixed parameters may include (a) the function transition boundary, (b) the thread number, (c) the instance count at which the stress may be applied and/or (d) the instance count at which the stress may be removed. In some embodiments, the tester may also indicate if the stress instance is a code exerciser stress, and/or a negative test, which tests a code block's handling of invalid input and/or unexpected behavior in regards to the code block's functionality (e.g., exception handling). If the stress is a negative stress (e.g., a stress associated with a negative test), then the tester may also specify the next transition that may occur for the stress test to pass and for the stress test to fail.

According to some embodiments, the following indicates how each variable parameter of the stress may be reflected in a list, such as the dashboard of FIG. 6. For injecting system resource stress into a targeted function transition, the tester may select a return value that represents a specific resource failure. For example, the tester may select an out of memory, out of disk space, socket disconnected, and/or file does not exist type of system resource error. For owned or third party application code stress, the tester may indicate the state at which stress should be applied. In some embodiments, state types may include: at the input, body, and/or end of the targeted function or API. For input parameter stress, where the data type is a simple type, the user may specify an alternate value. Likewise for simple return values, the user may specify an alternate return value. For complex data types, the tester may provide an alternate body of code. Some embodiments may compile and/or link computer code and/or inject it into the address space of the process in real time. When the targeted function is called and the count instance conditions are satisfied, the injected functionality will be called instead of the original functionality.

In some embodiments, each of the identified critical functionality may be subjected to a variety of stress. When the aforementioned list may be exhaustively tested, the next set of critical functionality may be targeted until gradually the last of the functionality is tested. In some embodiments, one advantage of the aforementioned real time code substitution mechanism is that it may also be used to return errors, which enables hard-to-reach independent code paths to get exercised as well.

Monitoring Agent and Analysis Engine Infrastructure

FIG. 7A depicts a high level block diagram of an example monitoring agent and analysis engine infrastructure. This infrastructure may be configured on a various hardware including computing devices ranging from smartphones, tablets, laptops, desktops to high end servers. As shown in this figure, data collection performed by the Monitoring Agent 702 may be segregated from analysis performed by the analysis Engine to improve application performance. The infrastructure provides high availability to prevent hackers from subverting its protection against malware attacks. The Monitoring Agent 702 interacts with an application to gather load time and runtime data. The infrastructure of the application 701 includes process memory 703, third-party libraries 704, kernel services 706, and an instruction pipeline 707. The infrastructure of the Monitoring Agent 702 includes the Instrumentation & Analysis Engine (instrumentation engine) 705, graphical user interface (GUI) 711, Client Daemon 708, Configuration database 709, and Streaming and Compression Engine 710, and central processing unit (CPU) 736. Local or remote users 738 of the application 701 interact with the application either through devices like keyboards, mice or similar I/O devices or over a network through a communication channel that may be established by means of pipes, shared memory or sockets. In response the application process 703 dispatches appropriate sets of instructions into the instruction pipeline 707 for execution. The application may also leverage its own or third party libraries 704 such as libc.so (Linux) or msvcrtxx.dll (Windows). As functionality from these libraries is invoked, appropriate instructions from these libraries are also inserted into the instruction pipeline for execution 707. In addition the application may leverage system resources such as memory, file I/O etc. from the kernel 706. These sequences of instructions from the application, libraries and the kernel put together in a time ordered sequence deliver the application functionality desired by a given user.

As the application's code begins to load into memory, the Instrumentation and Analysis Engine (i.e., instrumentation engine) 705 performs several different load time actions. Once all the modules have loaded up, the instrumented instructions of the application generate runtime data. The Client Daemon 708 initializes the Instrumentation and Analysis Engine 705, the Streaming Engine 710 and the GUI 711 processes in the CPU at 736 by reading one or more configuration files from the Configuration database 709. It also initializes intercommunication pipes between the instrumentation engine, Streaming Engine, GUI, Instrumentation & Analysis Engine 705 and itself. The Client Daemon also ensures that if any Monitoring Agent process, including itself, becomes unresponsive or dies, it will be regenerated. This ensures that the Monitoring Agent 702 is a high availability enterprise grade product.

The Instrumentation and Analysis Engine 705 pushes load and runtime data collected from the application into the Streaming Engine. The Streaming Engine packages the raw data from the Monitoring Agent 702 into the PDU. Then it pushes the PDU over a high bandwidth, low latency communication channel 712 to the Analysis Engine 728. If the Monitoring Agent 702 and the Analysis Engine 728 are located on the same machine this channel can be a memory bus. If these entities are located on different hardware but in the same physical vicinity, the channel can be an Ethernet or Fiber based transport, which allows remote connections to be established between the entities to transport the load and runtime data across the Internet.

The infrastructure of the Analysis Engine 728 includes the Network Interface Card (NIC) 713, the Packet Pool 714, the Time Stamp Engine 715, the Processor Fabric 716, the Hashing Engine 717, the TCAM Engine 718, the Application Map database 719, and the Thread Context database 720, which may contain a table of the memory addresses used by a class of user executing an application monitored by the system. The infrastructure of the Analysis Engine 728 further includes the Content Analysis Engine 721, the Events and Event Chains 722, the Event Management Engine 723, the Event Log 724, the Application Daemon 725, the Analysis Engine Configuration database 726, the Network Interface 727, the Dashboard or CMS 737, the SMS/SMTP Server 729, the OTP Server 730, the Upgrade Client 731, the Software Upgrade Server 732, Software Images 733, the Event Update Client 734, and the Event Upgrade Server 735.

The PDU together with the protocol headers is intercepted at the Network Interface Card 713 from where the PDU is pulled and put into the Packet Pool 714. The timestamp fields in the PDU are filled up by the Time Stamp Engine 715. This helps to make sure that no packet is stuck in the packet Pool buffer for an inordinately long time.

The Processor Fabric 716 pulls packets from the packet buffer and the address fields are hashed and replaced in the appropriate location in the packet. This operation is performed by the Hashing Engine 717. Then the Processor Fabric starts removing packets from the packet buffer in the order they arrived. Packets with information from the load time phase are processed such that the relevant data is extracted and stored in the Application Map database 719. Packets with information from the runtime phase are processed in accordance with FIG. 5. The efficiency of the Analysis Engine 728 can be increased or decreased based on the number of processors in the Processor Fabric.

The transition target data is saved in the Thread Context database 720 which has a table for each thread. The Processor fabric also leverages the TCAM Engine 718 to perform transition and memory region searches. Since the processor fabric performing lookups using hashes, the actual time used is predictable and very short. By choosing the number of processors in the fabric carefully, per packet throughput can be suitable altered.

When the Analysis Engine 728 performs searches, it may, from time to time find an invalid transition, invalid operation of critical/admin functions or system calls, or find a memory write on undesirable locations. In each of these cases, the Analysis Engine 728 dispatches an event of the programmed severity as described by the policy stored in the Event and Event Chain database 722 to the Event Management Engine 723. The raw event log is stored in the Event Log Database 724. The Dashboard/CMS 737 can also access the Event Log and display application status.

A remedial action is also associated with every event in the Event and Event Chain database 722. A user can set the remedial action from a range of actions from ignoring the event in one extreme to terminating the thread in the other extreme. A recommended remedial action can be recommended to the analyst using the Event Update Client 734 and Event Upgrade Server 735. In order to change the aforementioned recommended action, an analyst can use the Dashboard/CMS 737 accordingly. The Dashboard/CMS 737 provides a GUI interface that displays the state of each monitored application and allows a security analyst to have certain control over the application, such as starting and stopping the application. When an event is generated, the Event Chain advances from the normal state to a subsequent state. The remedial action associated with the new state can be taken. If the remedial action involves a non-ignore action, a notification is sent to the Security Analyst using and SMS or SMTP Server 729. The SMS/SMTP address of the security analyst can be determined using an LDAP or other directory protocol. The process of starting or stopping an application from the Dashboard/CMS 737 requires elevated privileges so the security analyst must authenticate using an OTP Server 730.

New events can also be created and linked into the Event and Event Chain database 722 with a severity and remedial action recommended to the analyst. This allows unique events and event chains for a new attack at one installation to be dispatched to other installations. For this purpose, all new events and event chains are loaded into the Event Upgrade Server 735. The Event Update Client 734 periodically connects and authenticates to the Event Upgrade Server 735 to retrieve new events and event chains. The Event Update Client then loads these new events and event chains into the Events and Events Chain database 722. The Content Analysis Engine 721 can start tracking the application for the new attacks encapsulated into the new event chains.

Just as with the Client Daemon, the Appliance Daemon 725 is responsible for starting the various processes that run on the Analysis Engine 728. For this purpose, it must read configuration information from the Analysis Engine Configuration database 726. The daemon is also responsible for running a heartbeat poll for all processes in the Analysis Engine 728. This ensures that all the devices in the Analysis Engine ecosystem are in top working condition at all times. Loss of three consecutive heartbeats suggests that the targeted process is not responding. If any process has exited prematurely, the daemon will revive that process including itself.

From time to time, the software may be upgraded in the Appliance host, or of the Analysis Engine 728 or of the Monitoring Agent 702 for purposes such as fixing errors in the software. For this purpose, the Upgrade Client 731 constantly checks with the Software Upgrade Server 732 where the latest software is available. If the client finds that the entities in the Analysis Engine 728 or the Monitoring Agent 702 are running an older image, it will allow the analysts to upgrade the old image with a new image from the Software Upgrade Server 732. New images are bundled together as a system image 733. This makes it possible to provision the appliance or the host with tested compatible images. If one of the images of a subsystem in the Analysis Engine 728 or the Monitoring Agent 702 does not match the image for the same component in the System image, then all images will be rolled to a previous known good system image.

PDU for Monitoring Agent and Analysis Engine Communication

FIG. 7B illustrates an example protocol data unit (PDU) used to transmit data between the Monitoring Agent 702 and an Analysis Engine 728 of FIG. 7A. In order for the Monitoring Agent 702 and the Analysis Engine 728 to work effectively with each other, they communicate with each other using the PDU. The PDU can specifically be used by the Monitoring Agent 702 to package the extracted model of the application and/or collected runtime data for transmission to the Analysis Engine 728. The PDU contains fields for each type of information to be transmitted between the Monitoring Agent 702 and the Analysis Engine 728. The PDU is divided into the Application Provided Data Section, the HW/CVE Generated, and Content Analysis Engine or Raw Data sections.

The Application Provided Data Section contains data from various registers as well as source and target addresses that are placed in the various fields of this section. The Protocol Version contains the version number of the PDU 752. As the protocol version changes over time, the source and destination must be capable of continuing to communicate with each other. This 8 bit field describes the version number of the packet as generated by the source entity. A presently unused reserved field 756 follows the Protocol Version field.

The next field of the Application Provided Data Section is the Message Source/Destination Identifiers 757, 753, and 754 are used to exchange traffic within the Analysis Engine infrastructure as shown in FIG. 7. From time to time, the various entities shown in FIG. 7, exchange traffic between themselves. Not all these devices have or need IP addresses and therefore, the two (hardware and host) Query Router Engines uses the Message Source and Destination fields to route traffic internally. Some messages need to go across the network to entities in the Analysis Engine. For this purpose, the entities are assigned the following IDs. A given Analysis Engine appliance may have more than one accelerator card. Each card will have a unique IP address; therefore, the various entities will have a unique ID. The aforementioned infrastructure may also be running more than one application. Since each application server will have a unique IP address, the corresponding Monitoring Agent side entity will also have a unique ID.

Monitoring Agent Side Entities

1. GUI

2. Instrumentation and Analysis Engine

3. Client Message Router

4. Streaming Engine

5. Client Side Daemon

6. CLI Engine

7. Client Watchdog

8. Client Compression Block

9. Client iWarp/RDMA/ROCE Ethernet Driver (100 Mb/1 Gb/10 Gb)

Per PCI Card Entities (Starting Address=20+n*20)

20. Analysis Engine TOE block

21. Analysis Engine PCI Bridge

22. Decompression Block

23. Message Verification Block

24. Packet Hashing Block

25. Time-Stamping Block

26. Message Timeout Timer Block

27. Statistics Counter Block

28. Analysis Engine Query Router Engine

29. Analysis Engine Assist

Analysis Engine Host Entities

200. Analysis Engine PCIe Driver

201. Host Routing Engine

202. Content Analysis Engine

203. Log Manager

204. Daemon

205. Web Engine

206. Watchdog

207. IPC Messaging Bus

208. Configuration Database

209. Log Database

SIEM Connectors

220. SIEM Connector 1—Dashboard/CMS

221. SIEM Connector 2—HP ArcSight

222. SIEM Connector 3—IBM QRadar

223. SIEM Connector 4—Alien Vault USM

Analysis Engine Infrastructure Entities

230. Dashboard/CMS

231. SMTP Server

232. LDAP Server

233. SMS Server

234. Entitlement Server

235. Database Backup Server

236. OTP Client

237. OTP Server

238. Checksum Server

239. Ticketing Server

240. Event Chain Upgrade Server

241. Software Update Server

All User Applications

255. User Applications—Application PID is used to identify the application issuing a query

Another field of the Application Provided Data section is the Message Type field which indicates the type of data being transmitted 755. At the highest level, there are three distinct types of messages that flow between the various local Monitoring Agent side entities, between the Analysis Engine appliance side entities and between Monitoring Agent side and appliance side entities. Furthermore, messages that need to travel over a network must conform to the OSI model and other protocols.

The following field of the Application Provided Data section is the Packet Sequence Number field containing the sequence identifier for the packet 779. The Streaming Engine will perform error recovery on lost packets. For this purpose it needs to identify the packet uniquely. An incrementing signed 64 bit packet sequence number is inserted by the Streaming Engine and simply passes through the remaining Analysis Engine infrastructure. If the sequence number wraps at the 64 bit boundary, it may restart at 0. In the case of non-application packets such as heartbeat or log message etc., the packet sequence number may be −1.

The Application Provided Data section also contains the Canary Message field contains a canary used for encryption purposes 761. The Monitoring Agent 702 and the Analysis Engine 728 know how to compute the Canary from some common information but of a fresh nature such as the Application Launch time, PID, the license string, and an authorized user name.

The Application Provided Data section additionally contains generic fields that are used in all messages. The Application Source Instruction Address 780, Application Destination Instruction Address 758, Memory Start Address Pointer 759, Memory End Address Pointer 760, Application PID 762, Thread ID 763, Analysis Engine Arrival Timestamp 764, and Analysis Engine Departure Timestamp 765 fields which hold general application data.

The PDU also contains the HW/CAE Generated section. In order to facilitate analysis and to maintain a fixed time budget, the Analysis Engine hashes the source and destination address fields and updates the PDU prior to processing. The HW/CAE Generated section of the PDU is where the hashed data is placed for later use. This section includes the Hashed Application Source Instruction Address 766, Hash Application Destination Instruction Address 767, Hashed Memory Start Address 768, and Hashed Memory End Address 769 fields. The HW/CAE Generated section additionally contains other fields related to the Canary 771 including the Hardcoded Content Start Magic header, API Name Magic Header, Call Context Magic Header and Call Raw Data Magic Header are present in all PDU packets.

The HW/CAE Generated section also includes a field 770 to identify other configuration and error data which includes Result, Configuration Bits, Operating Mode, Error Code, and Operating Modes data. The Result part of the field is segmented to return Boolean results for the different Analysis Engine queries—the transition playbook, the code layout, the Memory (Stack or Heap) Overrun, and the Deep Inspection queries. The Configuration Bits part of the field indicates when a Compression Flag, Demo Flag, or Co-located Flag is set. The presence of the flag in this field indicates to the Analysis Engine 728 whether the packet should be returned in compression mode. The Demo Flag indicates that system is in demo mode because there is no valid license for the system. In this mode, logs and events will not be available in their entirety. The Co-located Flag indicates that the application is being run in the Analysis Engine 728 so that Host Query Router Engine can determine where to send packets that need to return to the Application. If this flag is set, the packets are sent via the PCI Bridge, otherwise they are sent over the Ethernet interface on the PCI card. The Operating Mode part of the field indicates whether the system is in Paranoid, Monitor, or Learn mode. These modes will be discussed in more details later in this section. Lastly, the Error Code part of the field indicates an error in the system. The first eight bits of the error code will correspond to the message source. The remaining 12 bits will correspond to the actual error reported by each subsystem.

The PDU also contains the Content Analysis Engine or Raw Data. All variable data such as arguments and return value of the OS library calls and System Calls is placed in this section of the PDU. The data in this section contains the content of the data collected from the application and is primarily targeted at the Content Analysis Engine. This section contains the Variable Sized API Name or Number 772, the Call Content Magic Header 777, the Variable Sized Call Content 774, the Call Raw Data Magic Header 778, Variable Sized Raw Data Contents 776, and two reserved 773 and 775 fields. Furthermore, these fields can be overloaded for management messages.

Digital Processing Infrastructure

FIG. 8 illustrates a computer network or similar digital processing environment in which embodiments of the present disclosure may be implemented.

Client computer(s)/devices 50 and server computer(s) 60 provide processing, storage, and input/output devices executing application programs and the like. The client computer(s)/devices 50 can also be linked through communications network 70 to other computing devices, including other client devices/processes 50 and server computer(s) 60. The communications network 70 can be part of a remote access network, a global network (e.g., the Internet), a worldwide collection of computers, local area or wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth®, etc.) to communicate with one another. Other electronic device/computer network architectures are suitable.

Client computers/devices 50 may be configured with the monitoring agent. Server computers 60 may be configured as the analysis engine which communicates with client devices (i.e., monitoring agent) 50 for detecting database injection attacks. The server computers 60 may not be separate server computers but part of cloud network 70. In some embodiments, the server computer (e.g., analysis engine) may analyze a set of computer routines and identify one or more computer routines of the set having a likelihood of vulnerability. The client (monitoring agent, and/or in some embodiments a validation engine) 50 may communicate a manipulation of the computer routines through a testing technique to the server (analysis engine) 60. In some embodiments, the client 50 may include client applications or components (e.g., instrumentation engine) executing on the client (i.e., monitoring agent, and/or in some embodiments a validation engine) 50 for initiating tests to asynchronously and dynamically manipulate the computer routines and determine unexpected behavior of the computer routines, and the client 50 may communicate this information to the server (e.g., analysis engine) 60.

FIG. 9 is a diagram of an example internal structure of a computer (e.g., client processor/device 50 or server computers 60) in the computer system of FIG. 8. Each computer 50, 60 contains a system bus 79, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The system bus 79 is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements. Attached to the system bus 79 is an I/O device interface 82 for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer 50, 60. A network interface 86 allows the computer to connect to various other devices attached to a network (e.g., network 70 of FIG. 8). Memory 90 provides volatile storage for computer software instructions 92 and data 94 used to implement an embodiment of the present disclosure (e.g., monitoring agent, instrumentation engine, and analysis engine elements described herein). Disk storage 95 provides non-volatile storage for computer software instructions 92 and data 94 used to implement an embodiment of the present disclosure. A central processor unit 84 is also attached to the system bus 79 and provides for the execution of computer instructions.

Embodiments or aspects thereof may be implemented in the form of hardware (including but not limited to hardware circuitry), firmware, or software. If implemented in software, the software may be stored on any non-transient computer readable medium that is configured to enable a processor to load the software or subsets of instructions thereof. The processor then executes the instructions and is configured to operate or cause an apparatus to operate in a manner as described herein.

Some embodiments may transform the behavior and/or data of a set of computer routines by asynchronously and dynamically manipulating at least one of the computer routines through at testing technique. The testing technique may include modification of a value, input parameter, return value, or code body associated with one or more of the computer routines, thereby transforming the behavior (and/or data) of the computer routine.

Some embodiments may provide functional improvements to the quality of computer applications, computer program functionality, and/or computer code by detecting improper handling of error conditions and/or vulnerabilities in the computer applications and/or computer code by way of the testing techniques. Some embodiments may check to see if the control flow of a thread changed as a result of manipulation (e.g., fuzzing), by comparing the control flow extracted with and without the given computer routine being attacked (through an attack vector). Some embodiments may deploy a code path to correct and/or replace the computer routine to avoid the unexpected and/or incorrect behavior. As such, some embodiments may detect and correct computer code functionality, thereby providing a substantial functional improvement.

Some embodiments solve a technical problem (thereby providing a technical effect) of robustness of basic functionality of software and its error handling functionality. Some embodiments also solve a technical problem of exercising code paths that are too hard to reach in other test suites (thereby providing a technical effect). Some embodiments also provide a display to users in order to report status of testing, and thereby improve efficiency of testing, and thereby also solving a technical problem of lack of efficiency in test (and thereby also providing a technical effect).

Further, hardware, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions of the data processors. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

It should be understood that the flow diagrams, block diagrams, and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and, thus, the data processors described herein are intended for purposes of illustration only and not as a limitation of the embodiments.

While this disclosure has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure encompassed by the appended claims.

Claims

1. A computer-implemented method for determining unexpected behavior of a computer routine, the method comprising: analyzing a set of computer routines to identify a given computer routine of the set of computer routines having a vulnerability to unexpected behavior wherein, the identifying includes determining functions and system call sequences used by each thread associated with the set of computer routines to map run time control flow of the set of computer routines and the analysis includes at least one of: (i) performing a simulation to cause at least one failure condition and observing response of at least one computer routine of the set of computer routines to the simulation, (ii) extracting a histogram including a frequency of usage associated with the set of computer routines, (iii) determining size of one or more buffer read or write computer operations associated with the set of computer routines, (iv) determining size of one or more corresponding stacks associated with the set of computer routines, (v) determining size of one or more memory read or write operations based upon examining a corresponding loop size, and (vi) performing a taint analysis of one or more computer routine of the set;manipulating the given computer routine using a testing technique; anddetermining unexpected behavior of the given computer routine based on a result of the manipulating.

2. The method of claim 1 further comprising: deploying one or more patches to correct the unexpected behavior of the given computer routine.

3. The method of claim 1 further comprising: analyzing at least one corresponding sequence of computer routines of the set of computer routines.

4. The method of claim 1 wherein the given computer routine includes at least one of: a function; anda system call.

5. The method of claim 1 wherein manipulating the given computer routine further comprises at least one of: modifying data associated with the given computer routine, the data exceeding a corresponding buffer size; andmodifying values that are declared in memory regions associated with the given computer routine.

6. The method of claim 1, wherein determining the unexpected behavior of the given computer routine includes: determining that a control flow of a given thread associated with the given computer routine has changed as a result of the manipulation;determining a failure condition that caused the thread to change its control flow; anddisplaying the failure condition.

7. The method of claim 1, wherein, for a function of the given computer routine, the testing technique provides at least one of invalid, unexpected, and random data to at least one of an input of the function, logic within the function, and an output of the function.

8. The method of claim 1, wherein, for a system call of the given computer routine, the testing technique provides at least one of invalid, unexpected, and random data to a system call parameter associated with the system call.

9. The method of claim 8, wherein the system call parameter is associated with at least one of: thread synchronization, process synchronization, thread scheduling, process scheduling, memory, memory allocation, memory de-allocation, memory writing, memory reading, a network socket, creation of a network socket, network socket input, network socket output, pipe creation, system input, system output, shared memory fifo creation, a terminal input, a terminal output, file handling, file creation, file writing, file reading, disk input, and disk output.

10. A system for determining unexpected behavior of a computer routine, the system comprising: a processor; anda memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions, being configured to cause the system to:analyze a set of computer routines to identify a given computer routine of the set of computer routines having a vulnerability to unexpected behavior wherein, the identifying includes determining functions and system call sequences used by each thread associated with the set of computer routines to map run time control flow of the set of computer routines and the analysis includes at least one of: (i) performing a simulation to cause at least one failure condition and observing response of at least one computer routine of the set of computer routines to the simulation, (ii) extracting a histogram including a frequency of usage associated with the set of computer routines, (iii) determining size of one or more buffer read or write computer operations associated with the set of computer routines, (iv) determining size of one or more corresponding stacks associated with the set of computer routines, (v) determining size of one or more memory read or write operations based upon examining a corresponding loop size, and (vi) performing a taint analysis of one or more computer routine of the set;manipulate the given computer routine using a testing technique; anddetermine unexpected behavior of the given computer routine based on a result of the manipulating.

11. The system of claim 10 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to: deploy one or more patches to correct the unexpected behavior of the given computer routine.

12. The system of claim 10 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to: analyze at least one corresponding sequence of computer routines of the set of computer routines.

13. The system of claim 10 wherein the given computer routine includes at least one of: a function; anda system call.

14. The system of claim 10 where, in manipulating the given computer routine, the processor and the memory, with the computer code instructions, are further configured to cause the system to: modify data associated with the given computer routine, the data exceeding a corresponding buffer size; andmodify values that are declared in memory regions associated with the given computer routine.

15. The system of claim 10 where, in determining the unexpected behavior of the given computer routine, the processor and the memory, with the computer code instructions, are further configured to cause the system to: determine that a control flow of a given thread associated with the given computer routine has changed as a result of the manipulation;determine a failure condition that caused the thread to change its control flow; anddisplay the failure condition.

16. The system of claim 10, wherein, for a function of the given computer routine, the testing technique provides at least one of invalid, unexpected, and random data to at least one of an input of the function, logic within the function, and an output of the function.

17. The system of claim 10, wherein, for a system call of the given computer routine, the testing technique provides at least one of invalid, unexpected, and random data to a system call parameter associated with the system call.

18. A non-transitory computer program product for determining unexpected behavior of a computer routine, the computer program product comprising a computer-readable medium with computer code instructions stored thereon, the computer code instructions being configured, when executed by a processor, to cause an apparatus associated with the processor to: analyze a set of computer routines to identify a given computer routine of the set of computer routines having a vulnerability to unexpected behavior wherein, the identifying includes determining functions and system call sequences used by each thread associated with the set of computer routines to map run time control flow of the set of computer routines and the analysis includes at least one of: (i) performing a simulation to cause at least one failure condition and observing response of at least one computer routine of the set of computer routines to the simulation, (ii) extracting a histogram including a frequency of usage associated with the set of computer routines, (iii) determining size of one or more buffer read or write computer operations associated with the set of computer routines, (iv) determining size of one or more corresponding stacks associated with the set of computer routines, (v) determining size of one or more memory read or write operations based upon examining a corresponding loop size, and (vi) performing a taint analysis of one or more computer routine of the set;manipulate the given computer routine using a testing technique; anddetermine unexpected behavior of the given computer routine based on a result of the manipulating.