Patent Grant
9274926
This description relates to the configurable testing of computer programs.
Techniques for testing computer programs in source code form are well known. For example, compilers that transform source code written in a programming language into a computer usable form (e.g., binary code) often test the source code and alert software developer(s) to any issues that may render the code un-executable.
In one aspect, in general, a computing system for configurable testing of a computer program includes: a storage medium storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and a configurable testing module, including at least one processor, configured to process the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.
Aspects can include one or more of the following features.
The processing further includes: performing at least one test specified in at least the first testing specification based at least in part on stored values of attributes of one or more recognized portions of the computer program.
The first testing specification includes a test that tests at least one characteristic of the computer program that does not prevent the computer program from executing without error.
The first testing specification includes a test that tests at least one characteristic of the computer program associated with code reuse efficiency.
The first testing specification includes a test that tests at least one characteristic of the computer program associated with memory use constraints.
The characteristic of the computer program associated with memory use constraints includes a maximum amount of main memory that a particular recognized portion of the computer program is allowed to use when the computer program executes.
The first testing specification includes a test that tests a number of times at least one particular operation is used in the computer program.
The first testing specification includes a test that references a value of a name attribute of a recognizable portion of the computer program to determine whether to continue applying the test.
The processing further includes presenting in a user interface a result of at least one test, the result being visually associated with a displayed portion of the computer program on which the test was performed.
The storage medium stores activation information indicating which of the testing specifications are active.
At least one of the testing specifications includes at least one local test that applies to a particular portion of the computer program, and at least one of the testing specifications includes at least one global test that applies to multiple portions of the computer program.
The processing further includes: accessing the activation information to determine one or more testing specifications that are indicated as active.
The set of one or more testing specifications is a hierarchical set that relates at least some of the testing specifications to each other according to a hierarchy.
Accessing the activation information to determine one or more testing specifications that are indicated as active includes identifying active testing specifications along a path up the hierarchy between the first testing specification and a root of the hierarchy.
The processing further includes: while traversing the representation, performing any local tests specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.
The processing further includes: while traversing the representation, performing any aggregation functions specified in any active testing specifications based at least in part on stored values of attributes of one or more recognized portions of the computer program.
The processing further includes: after traversing the representation, performing any global tests specified in any active testing specifications based at least in part on stored results of any aggregation functions performed while traversing the representation.
The representation of the computer program is a graph based-representation and the elements include: nodes that represent at least some of the recognizable portions, and links that represent relationships between the recognizable portions represented by the nodes.
The graph-based representation includes a dataflow graph that includes: processing components represented by the nodes, and flows of data between processing components represented by the links.
At least some of the recognizable portions include a flow of data between processing components.
At least some of the recognizable portions include an input or output port of a processing component.
At least some of the recognizable portions include a parameter associated with a processing component or a flow of data between processing components.
The graph-based representation includes a control flow graph that includes: basic blocks of code represented by the nodes, and changes of control between basic blocks represented by the links.
The testing further including receiving user input for specifying at least one testing specification of the set of one or more testing specifications.
At least some of the attribute information is predetermined before at least some of the user input is received.
In another aspect, in general, a computing system for configurable testing of a computer program includes: means for storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and means for processing the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.
In another aspect, in general, a method for configurable testing of a computer program includes: storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and processing, using at least one processor, the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.
In another aspect, in general, software is stored on a computer-readable medium, for configurable testing of a computer program. The software includes instructions for causing a computing system to: store a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and process the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.
Aspects can have one or more of the following advantages.
A company, or other organization, may have groups of software developers that develop computer programs for performing various tasks (e.g., processing company or customer data, or processing transactions). While compilers, debugging systems, or other core error testing systems are able to test for correctness of the programs being developed, it may be useful to enable a testing system to be extended in flexible ways to test for specific conventions or programming style that different groups in the organization may wish to follow. The techniques described herein enable the definition of configurable testing extensions that can be customized for different groups (or for different branches of a hierarchical organization, as described in more detail below). To facilitate such flexible extension of the testing system, the configurable testing system is configured to collect information about various attributes of recognizable portions of a computer program so that tests can be defined for those attributes. For example, in an organization that develops programs for processing data, there may be portions of a program that correspond to the movement of data from one program module to another, and portions of a program that correspond to data processing functions or data transformations. The collected information about these portions of the program enables a user to define a test relating to the functional structure or efficiency, for example, of a computer program (e.g., to recognize certain uses or arrangements various program modules that may be inefficient or incompatible with known conventions). These tests enable the development environment to provide visual indications to a developer automatically about various undesirable conditions that may prevail in the computing system that will be executing the program being developed. For example, for programs that process relatively large amounts of data in a relatively short amounts of time, certain technical conditions of the programmed computing system that effect computation efficiency can be automatically flagged to the developer at the time the program is being developed. Those conditions may be identified based on characteristics of the computer program, even if those characteristics would not prevent the computer program from executing without error. Solution of the technical problem of automatically identifying those characteristics is facilitated by providing stored information about attributes of recognizable portions of the computer program.
Other features and advantages of the invention are apparent from the following description, and from the claims.
1 System Overview
The execution environment 104 may be hosted, for example, on one or more general-purpose computers under the control of a suitable operating system, such as a version of the UNIX operating system. For example, the execution environment 104 can include a multiple-node parallel computing environment including a configuration of computer systems using multiple central processing units (CPUs) or processor cores, either local (e.g., multiprocessor systems such as symmetric multi-processing (SMP) computers), or locally distributed (e.g., multiple processors coupled as clusters or massively parallel processing (MPP) systems, or remote, or remotely distributed (e.g., multiple processors coupled via a local area network (LAN) and/or wide-area network (WAN)), or any combination thereof.
In general, the execution environment 104 is configured to read data from the data source 102 and to execute a computer program 114 to apply a data processing algorithm to the data to generate output data. That output data may be stored back in the data source 102 or in the data storage system 116, or otherwise used. In some examples, the computer program 114 is implemented as a dataflow graph that includes vertices (representing data processing components or datasets) connected by directed links (representing flows of work elements, i.e., data) between the vertices. For example, such an environment is described in more detail in U.S. Publication No. 2007/0011668, titled “Managing Parameters for Graph-Based Applications,” incorporated herein by reference. A system for executing such graph-based computations is described in U.S. Pat. No. 5,566,072, titled “EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS,” incorporated herein by reference. Dataflow graphs made in accordance with this system provide methods for getting information into and out of individual processes represented by graph components, for moving information between the processes, and for defining a running order for the processes. This system includes algorithms that choose interprocess communication methods from any available methods (for example, communication paths according to the links of the graph can use TCP/IP or UNIX domain sockets, or use shared memory to pass data between the processes).
Storage devices providing the data source 102 may be local to the execution environment 104, for example, being stored on a storage medium connected to a computer hosting the execution environment 104 (e.g., hard drive 108), or may be remote to the execution environment 104, for example, being hosted on a remote system (e.g., mainframe 110) in communication with a computer hosting the execution environment 104, over a remote connection.
The execution environment 104 can receive data from a variety of types of systems that may embody the data source 102, including different forms of database systems. The data may be organized as records having values for respective fields (also called “attributes” or “columns”), including possibly null values. When first reading data from a data source, the execution environment 104 typically starts with some initial format information about records in that data source. In some circumstances, the record structure of the data source may not be known initially and may instead be determined after analysis of the data source or the data. The initial information about records can include, for example, the number of bits that represent a distinct value, the order of fields within a record, and the type of value (e.g., string, signed/unsigned integer) represented by the bits.
The data storage system 116 is also accessible to the development environment 118, which enables a developer 120 to develop the dataflow graphs described above, including specifying vertices (representing components or datasets) connected by directed links (representing flows of work elements) between the vertices. In some examples, the development environment 118 is configured to facilitate configurable testing of the dataflow graph specified by the developer 120, including specifying and applying user-defined extension tests, as is described in greater detail below.
2 Configurable Testing Examples
Referring to
Some components such as the ‘Rollup’ component 226 and the ‘Sort’ component 230 also include input parameters 240 which are used to configure the functionality of the components 226, 230. For example, an input parameter may include a reference to a value that will be supplied at runtime when the dataflow graph 200 is being prepared for execution.
During development of the dataflow graph 200, one or more developers may inadvertently introduce different types of errors. For example, in some instances, when specifying (i.e., constructing) the graph 200 using the development environment 118, a developer 120 may introduce one or more errors that will prevent the graph 200 from running. For example, the developer 120 may neglect to specify one or more of the input parameters 240 to the components, the developer may neglect to connect a flow 234 to all of the required input ports 236 of a given component, or the developer 120 may connect an incorrect data type to an input port 236 of a component. In other examples, the developer 120 could specify a graph 200 that, while technically functional, does not conform to style guidelines and/or best practices for graph development. Thus, there is a need to test the graphs specified by developers to inform the developers about different programming style preferences or to warn the developers of the presence of different types of inefficiencies, in addition to warning the developers of presence of different types of errors or identifying characteristics that would prevent the computer program from executing without error.
Referring to
The result of the configurable testing module 342 walking through the dataflow graph 200 is a testing output 348. The testing output 348 may include any testing messages that were triggered by applying the various tests to the dataflow graph 200 (e.g., warning messages based on failure of an applied test). The testing output 348 is presented in a user interface to the developer 120 by the user interface module 350. The user interface module 350 can present the testing failures in a way that visually associates each failure with the relevant part of the dataflow graph (or relevant part of the source code or other representation of a computer program). In general, the developer 120 views the testing output 348 and modifies the specification of the dataflow graph 200 if necessary.
2.1 Error Identification Tests
The set of error identification tests 344 includes tests 345 that are designed to alert the developer 120 to serious errors in the specification of the dataflow graph 200. For example, if the developer 120 neglected to connect flows to the input ports of a component in the dataflow graph, the dataflow graph 200 would likely not run at all (i.e., it would not produce useable output and/or execution would halt at some point short of completion). The application of the set of error identification tests 344 by the configurable testing module 342 would alert the developer to such an error, indicating that the error needs to be fixed before the dataflow graph 200 will run. Such error identification tests include tests for syntax errors and other correctness tests that are common in software development environments, and therefore will not be discussed further in this description.
2.2 Extension Tests
Each extension test 347 of the set of extension tests 346 includes one or more user-specified extension procedures that define at least one test or function that can be applied to the dataflow graph 200, such as for the purpose of enforcing company specific styles (e.g., naming conventions, use of subgraphs, etc.), enforcing best practices, and to warn of common dataflow graph development pitfalls.
The developer 120 is able to include in these procedures attributes of any recognizable portions of the dataflow graph (e.g., the data structures listed below for various dataflow graph objects), as defined in stored attribute information 352. For example, components of a dataflow graph may have attributes that identify input and output ports of the component, the type of the component, and other relevant metadata; and flows of a dataflow graph may have attributes that identify upstream and downstream ports and components to which the flow is connected, the type of work elements being transferred along the flow, and other relevant metadata.
In some examples, there are three types of extension procedures for defining, respectively, simple tests, aggregation functions, and aggregation tests. In general, the type of extension procedure is determined by the return type of the defined test or function. For example, a simple test can be defined by writing a procedure with a return type of vld_simple_result_t, an aggregation function can be defined by writing a procedure with a return type of vld_aggr_global_result_t, vld_aggr_global_from_vec_result_t, or vld_aggr_local_result_t, and an aggregation test can be defined by writing a procedure with a return type of vld_check_aggr_result_t.
The configurable testing module 342 evaluates the extension procedures based on values of attributes of recognizable portions of a program such as the dataflow graph 200, in multiple stages. For example, in a first stage, the configurable testing module 342 traverses the dataflow graph 200 and applies the simple tests and the aggregation functions to recognizable objects as the configurable testing module 342 traverses the graph. For example, for each object that the configurable testing module 342 visits while traversing the graph, values of attributes of the object are packaged into a data structure. The data structure is then passed to either a simple test or an aggregation function (both described in detail below), which operates on the data structure. The configurable testing module 342 can traverse the flows and components of the dataflow graph, as well as the ports connecting the flows to the components, in any order. Traversing such parts of a dataflow graph or other computer program enables the configurable testing module 342 to locate each part once within an order that is pre-determined or determined as part of a search algorithm. For example, each part can be located once according to a depth-first search or a breadth-first search). In a second stage, after all of the simple tests and aggregation functions have run, the aggregation tests are run (described in more detail below). The aggregation tests operate on data stores (e.g., lookup tables) populated by the aggregation functions in the first stage. The aggregation tests are therefore able to test global properties of the dataflow graph 200 based on values aggregated from attributes of multiple objects in the graph.
2.2.1 Simple Tests
In general, simple tests run on a single type of construct in a computer program that is being tested (e.g., an expression, statement, or procedure in source code of a computer program, or a component, flow, or port of a computer program expressed as a dataflow graph) and are simple in the sense that they do not require information about other objects in the dataflow graph 200 to be evaluated.
As is noted above, when specifying a test, the developer 120 can designate the test as a simple test by setting the return type of the test to a predefined return type that is common to all simple tests (e.g., vld_simple_result_t). Simple tests receive a single argument, which is a data structure of attribute values specific to an associated object. In some examples, the data structure is a read-only structure that cannot be altered by the test. In some examples, each type of object is associated with a different type of data structure, as defined by the attribute information 352. For example, some different types of data structures that can be passed as arguments to simple tests for different types of objects are:
validation_parameter_t—Parameter data structure, associated with a parameter object, may include the following attribute values: parameter name, parameter type, value (of a resolved parameter), definition (of an unresolved parameter), whether the parameter is required or optional.
validation_component_t—Component data structure, associated with a component object, may include the following attribute values: component name, component type, presence of an input/output dataset coupled to an input/output port of the component.
validation_flow_t—Flow data structure, associated with a flow object, may include the following attribute values: flow name, upstream/downstream ports or components connected to the flow, whether the flow transmits encrypted data, whether the flow transmits compressed data, whether the flow is buffered.
validation_port_t—Port data structure, associated with a port object, may include the following attribute values: port name, dataset or component that contains the port, metadata associated with the port defining the type of data received or provided by the port, whether the port is connected to a flow.
validation_graphinfo_t—Graphinfo data structure, associated with a dataflow graph object, may include information about the dataflow graph as a whole.
Simple tests perform a user-defined test on the information included in the data structure. If the information in the data structure fails the user-defined test, the simple test emits an error indication.
In operation, as the configurable testing module 342 traverses the dataflow graph 200, it recognizes each of the objects in the graph and processes that object accordingly to determine values of various attributes that are defined according to the stored attribute information 352. When the configurable testing module 342 reaches a given type of object, it refers to the attribute information 352 to determine what attributes an object of that type has, and it stores values of those attributes in the appropriate data structure for the given object. The configurable testing module 342 also determines whether there is a simple test associated with the given object. If so, the configurable testing module 342 evaluates the simple test, if necessary passing as an argument information about the given object stored the data structure that may be needed to evaluate the simple test. If the simple test for the given object determines that the attribute values associated with the given object violate the simple test, then the simple test returns an error status, which is accumulated by the configurable testing module 342 such that it can be included in the testing output 348 upon completion of the testing process.
2.2.1.1 Exemplary Simple Test
Referring to
Referring to line 1 of
The body of the test begins at line 2 with the keyword begin. At line 3, a variable named problem is created and initialized to 0. Line 4 is a conditional statement which sets the problem variable to 1 if the source component type of the flow (flow.source_component_type) is an Input File component (Input_file.mdc) and the target component type of the flow (flow.target_component_type) is a Replicate component (Replicate.mpc).
Line 6 includes a conditional statement which causes the extension test to exit without issuing any output if the problem variable is equal to 0. Otherwise, if the problem variable is equal to 1 (indicating that a Replicate component does follow an Input File component), then lines 8-11 are executed, causing the following warning message to be issued to the developer 120:
“Don't put a Replicate component directly after an Input File component. Instead, connect multiple flows directly to the Input File component”
The extension test ends at line 12.
2.2.2 Aggregation Functions
Aggregation functions do not emit error messages but instead exist for the purpose of aggregating information from objects in the dataflow graph, or from the dataflow graph as a whole, as the configurable testing module 342 traverses the dataflow graph. After the configurable testing module 342 finishes walking through the graph, the aggregated information is used by the aggregation tests described below. In some examples, there are two main types of aggregation functions: local aggregation functions and global aggregation functions. Examples of extension testing may use either or both as appropriate.
In general, local aggregation functions aggregate information from individual objects in the dataflow graph and global aggregation functions aggregate information from the dataflow graph as a whole. When specifying an aggregation function, the developer can designate the function as a local aggregation function by setting the return type to vld_aggr_local_result_t. Similarly, the developer can designate a function as a global aggregation function by setting the return type to one of: vld_aggr_global_result_t or vld_aggr_global_from_vec_result_t.
In some examples, as the configurable testing module 342 evaluates an aggregation function while traversing the dataflow graph, it populates one of two lookup tables based on its aggregation function type (i.e., local or global). In particular, a first lookup table of the two lookup tables is populated with information from local aggregation functions and a second lookup table of the two lookup tables is populated with information from global aggregation functions. In some examples, aggregated information is stored as vectors of strings in the lookup tables.
2.2.2.1 Exemplary Local Aggregation Function
Referring to
At line 1 of
The body of the function begins at line 3 with the keyword begin. At lines 4-6, a conditional statement tests whether the port which was passed in as an argument belongs to a Reformat component (port.owner_type==“Reformat.mpc”) and is an input port (port.port_generic_name==“in”) or an output port (port.port_generic_name==“out”). If the above test condition is not satisfied, then the local aggregation function exits without aggregating any information into the local aggregation lookup table. Otherwise, if the above test condition is satisfied, then the resolved metadata for the port (port.resolved_metadata) is aggregated into a vector of strings in the local aggregation lookup table. The vector of strings is accessible by specifying two “keys”: “record formats,” and the full name of the Reformat component (port.owner_full_name). The extension function ends at line 11.
2.2.2.2 Exemplary Global Aggregation Functions
Referring to
Referring to lines 1-2 of
The body of the function begins at line 3 with the keyword begin. At line 4, a conditional statement tests whether the type of the input component argument (component.component_type) is Sort.mpc (i.e., is the component a Sort component). If the test indicates that the input component argument is of a type other than Sort.mpc, the aggregation function exits at line 5 without aggregating anything into the global aggregation lookup table. Otherwise, if the input component argument is of the Sort.mpc type, then the name of the input component argument (component.full_name) is written to the global aggregation lookup table at line 6 and is associated with the sort_components key in the lookup table at line 7. The extension function ends at line 8.
Referring to
Referring to lines 1-2 of
The body of the function begins at line 3 with the keyword begin. At line 4, a conditional statement tests whether the input component argument is a graph component (component.is_graph). If the test indicates that the input component argument is not a graph component, the aggregation function exits at line 5 without aggregating anything into the global aggregation lookup table. Otherwise, if the input component argument is a graph component, then the name of the input component argument (component.full_name) is written to the global aggregation lookup table at line 6 and is associated with the base_components key in the lookup table at line 7. The extension function ends at line 8.
2.2.3 Aggregation Tests
In general, aggregation tests operate on the information collected by the aggregation functions described above and are capable of emitting an error message if a problem is discovered.
As is noted above, when specifying a test, the developer 120 can designate the test as an aggregation test by setting the return type of the test to a predefined return type which is common to aggregation tests (e.g., vld_check_aggr_result_t). Aggregation tests access the global and local aggregation lookup tables (populated by the aggregation functions described above) for data that is used to evaluate a user-defined test. Some aggregation tests do not require any external arguments or information other than the information aggregated in the lookup tables, and some aggregation tests do optionally accept external arguments to be used in evaluating a test. In some examples, helper functions are provided to the developer 120 to facilitate retrieving information from the lookup tables. For example, the functions:
vld_lookup_by_key(key) and
vld_lookup_by_obj_key(obj, key)
can be used to retrieve a vector of strings corresponding to a key, or to an object name and a key, respectively.
As is noted above, aggregation tests are used to discover potential issues in a program that would not stop the program from running but may be indicative of a problem that should be investigated. In some examples, an aggregation test may alert the developer to a potential problem, prompting the developer to modify the program to avoid the problem. In other examples, the aggregation test may alert the developer to a potential problem, which, upon investigation, the developer deems to be an intended behavior of the program. For example, if an aggregation test determines that the number of Sort components in a dataflow graph exceeds a predetermined limit, it will alert the developer to the issue. It is up to the developer to decide whether the number of Sort components is, in fact, excessive.
2.2.3.1 Exemplary Aggregation Test
Referring to
Referring to line 1 of
The body of the test begins at line 2 with the keyword begin. At line 3, a vector of strings associated with the key base_components is read from the global aggregation lookup table using the command vld_lookup_by_key(“base_components”) and assigned to a variable named components. As is illustrated in
At line 5, the total number of components in the dataflow graph is determined by calculating the length of the components vector of strings (i.e., length_of(components)). The total number of components in the dataflow graph is stored in the how_many_base_components variable. Similarly, the at line 6, the number of Sort components in the dataflow graph is determined by calculating the length of the sorts vector of strings (i.e., length_of(sorts)). The number of Sort components in the dataflow graph is stored in the variable how_many_sorts variable.
At line 7, the total number of base components (i.e., how_many_base_components) is multiplied by 0.2 and assigned to a variable named limit. At line 8, a conditional statement applies a test to determine if the total number of Sort components in the dataflow graph (i.e., how_many_sorts) is less than or equal to the limit variable. If so, the aggregation test exits at line 9 without issuing an error, indicating that a reasonable number of Sort components are included in the dataflow graph. Otherwise, the aggregation test outputs the following error message at lines 10-12:
“Number of Sort components is greater than <limit>”
The extension test ends at line 13.
2.2.4 Extension Test Hierarchy
In some examples different departments within a company develop different projects that involve different computer programs (e.g., different dataflow graphs) or different portions of a program (e.g., one or more individual objects of a dataflow graph). The departments and/or the projects may have hierarchical relationships according to a development structure. Such a hierarchical development structure can be illustrated as an directed graph (e.g., a tree) or other hierarchical representation.
Referring to
Different testing specifications including extension procedures defining extension tests (and functions) can be specified for and associated with any of the nodes in the tree 951 to ensure that the individual portions of the dataflow graph (or other computer program) conform to specified style guidelines, best practices, etc. In the tree 951 of
The company level node 952 is the only node at its level and can be associated with a set of testing specifications that can be used to define a shared set of extension tests. For example, the set of extension tests associated with the company level node 952 will be selectively applied (if activated) to all departments and projects whose nodes are descendants of (i.e., can trace a path over parent-child relationships back to) the company level node 952. For example, a extension test to ensure that all components of a dataflow graph have particular characteristics may be specified for the company level node. All nodes that descend from the company level node 952 are able to access and apply any extension tests defined in testing specifications associated with the company level node 952 or any testing specifications along a path up the tree (or other hierarchy) between that node and the company level node 952. In some examples, extension tests that were specified in a higher level of the tree 951, but not activated there, can be selectively activated separately for any node at any lower level. Any of the department level nodes 958 can also be associated with a set of testing specifications. A set of testing specifications associated with a given department level node can be selectively applied to all projects depending from the given department level node. The extension tests defined in testing specifications associated with all nodes that the given department level node descends from (in this example, the company level node 952) can also be selectively applied to all projects depending from the given department level node.
Any of the project level nodes 960 can also be associated with a set of testing specifications. A set of testing specifications associated with a given project level node can be selectively applied only to the associated project. The extension tests defined in testing specifications associated with all nodes that the given project level node descends from can also be selectively applied to the project.
In some examples, each of the nodes in the tree 951 is associated with an environment (e.g., a file or directory, or other form of dedicated storage space such as a sandbox) that is specific to the node. The testing specifications associated with a node can be stored in the environment along with the computer program being tested (e.g., a specification of a dataflow graph and related files). For example, a single file of source code (e.g., called extensions.xfr) including the definitions of all of the extension tests associated with the node can be stored in the environment.
In some examples, an activation file (e.g., extensions.act) can also be stored in the environment for enabling or disabling individual extension tests. The activation file includes can include a listing of all of the extension tests that are activated for the node. For example, the activation file may include a line with a format such as:
In some examples, extension tests are defined using a high level programming language in a file with an .xfr extension. Suitable high level programming languages are C, C++, Java, or proprietary high level programming languages such as Ab Initio's DML programming language.
In some examples, activation files are defined using a high level programming language and are stored as a text file or data file, and have a particular extension (e.g., a .act extension). Suitable high level programming languages are C, C++, Java, XML, or proprietary high level programming languages such as Ab Initio's DML programming language.
In some examples, the file including the source code of the extension tests (i.e., the .xfr file) and the activation file (i.e., the .act file) are stored in the same environment as the graph definition files.
In some examples, the file including the source code of the extension tests (i.e., the .xfr file) imports (i.e., includes) an attribute information file, which defines a number of attributes of recognizable portions of a dataflow graph (or other computer program) that are necessary for defining extension tests. The attributes may include, for example, attributes that enable the specification of tests relating to code reuse efficiency, to memory use constraints, to a number of times that a particular operation is used in a computer program, or to other attributes that may affect efficiency or other programming style preferences.
An example of a recognizable portion of a dataflow graph that enables specification of a test relating to memory use constraints is a parameter. In particular, a parameter called “max_memory” may be associated with a particular component that specifies a maximum amount of main memory that the component is allowed to use when it executes. If that component has used the maximum amount of main memory during execution and still needs more storage space then it would need to use alternative storage (e.g., storage in a hard drive or flash drive serving as a backing store to the main memory). The attribute information 352 would store information defining a parameter name attribute and a parameter definition attribute, which indicate to the configurable testing module 342 that when a parameter object is recognized when traversing a dataflow graph, its parameter name and definition (among other attributes) are to be stored (e.g., within the data structure “validation_parameter_t”) to be available for aggregation functions defined in an extension tests file. A simple test or an aggregation test may determine whether the name of a recognized parameter is “max_memory” by including a reference to the parameter name attribute. The module 342 determines whether the value of the parameter name attribute is “max_memory” to determine whether to continue applying the test. For example, the rest of the test may ensure that that particular instance of the max memory parameter has a definition that satisfies certain requirements (e.g., it is defined in terms of another parameter, as opposed to being a constant value).
In general, the order of the extension tests in the .xfr file and the .act file does not necessarily correspond to the order in which the extension tests execute. Rather, the order of execution is determined by the configurable testing module 342 as it traverses the computer program or dataflow graph.
In general, when defining extension tests, a limited amount of information about the data structures that implement the dataflow graph is provided to the developer in the form of attribute information (described above). The attribute information may simplify the view of the (often complex) data structures implementing the dataflow graph for the developer. Also, since the testing system is able to traverse over all of the recognizable parts of a program (i.e., locating each recognizable part a single time according to some order), invoking the relevant extension procedures on each part, it can directly inform the user interface module 350 of testing failures, for display to the user within the appropriate context. For example, a port-level testing failure can be visually associated with the port of the component in the dataflow graph to which it applies. Since the user-written extension procedures can be written in terms of the recognizable program parts, the developers of the extension procedures are not burdened with providing code to locate and recognize those parts themselves.
In some examples, the development module includes a number of simple, commonly used extension tests and functions, which can be used as templates for development of more complex extension tests and functions.
The extension tests and functions described above are simple examples of extension tests and functions, which are presented to facilitate a general understanding of extension tests and functions. However, it is noted that an innumerable number of extension tests and functions are possible.
In some examples all extension tests and functions can be enabled or disabled using a single, master control. For example, an extension switch such as “—testing-extensions” may be specified when instructing the configurable testing module 342 to perform extension testing, the presence of the switch enabling the extension tests and functions. If the extension tests and functions are disabled, only the error identification tests are applied.
The above description includes examples using a dataflow graph representation of a computer program. However, it is noted that the techniques described above are not limited to a dataflow graph representation of a computer program and can be applied to any representation of a computer program that includes elements that represent recognizable portions of the computer program (e.g., a control flow graph that includes basic blocks of code represented by nodes, and changes of control between basic blocks represented by links between the nodes). A recognizable portion of the computer program can be a portion of the computer program that is recognizable based on its specification as a distinct component, module, or block of code that is able to communicate with (e.g., by transfer of data and/or control information) another distinct component, module, or block of code. A recognizable portion of the computer program can also be a portion of the computer program that is recognizable based on its specification as a means of communication (e.g., a data link or communication channel) between such distinct components, modules, or blocks of code.
4 Implementations
The testing approach described above can be implemented using a computer executing suitable software. For instance, the software implements procedures in one or more computer programs that execute on one or more programmed or programmable computer systems (which may be of various architectures such as distributed, client/server, or grid) each including at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. The software may form one or more modules of a larger program, for example, that provides other services related to the design and configuration of dataflow graphs. The nodes and elements of the graph can be implemented as data structures stored in a computer readable medium or other organized data conforming to a data model stored in a data repository.
The software may be provided on a tangible, non-transitory storage medium, such as a CD-ROM, readable by a general or special purpose programmable computer, or delivered (encoded in a propagated signal) over a communication medium of a network to a tangible, non-transitory storage medium of the computer where it is executed. Alternatively, some or all of the functions may be performed on a special purpose computer, or using special-purpose hardware, such as coprocessors or field-programmable gate arrays (FPGAs) or dedicated, application-specific integrated circuits (ASICs). The processing may be implemented in a distributed manner in which different parts of the computation specified by the software are performed by different computing elements. Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a tangible, non-transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform one or more of the functions described herein.
A number of embodiments of the invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. For example, some of the steps described above may be order independent, and thus can be performed in an order different from that described.
It is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the appended claims. Other embodiments are within the scope of the following claims.
This application is a continuation-in-part of and claims priority to U.S. application Ser. No. 13/733,403, filed on Jan. 3, 2013, incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3662343 | Goldstein et al. | May 1972 | A |
| 3662401 | Collins et al. | May 1972 | A |
| 4922418 | Dolecek | May 1990 | A |
| 4972314 | Getzinger et al. | Nov 1990 | A |
| 5127104 | Dennis | Jun 1992 | A |
| 5276899 | Neches | Jan 1994 | A |
| 5280619 | Wang | Jan 1994 | A |
| 5301336 | Kodosky | Apr 1994 | A |
| 5323452 | Dickman et al. | Jun 1994 | A |
| 5333319 | Silen | Jul 1994 | A |
| 5357632 | Pian et al. | Oct 1994 | A |
| 5418888 | Alden | May 1995 | A |
| 5432942 | Trainer | Jul 1995 | A |
| 5495590 | Comfort et al. | Feb 1996 | A |
| 5630047 | Wang | May 1997 | A |
| 5692168 | McMahan | Nov 1997 | A |
| 5701400 | Amardo | Dec 1997 | A |
| 5712971 | Stanfill et al. | Jan 1998 | A |
| 5745778 | Alfieri | Apr 1998 | A |
| 5799266 | Hayes | Aug 1998 | A |
| 5802267 | Shirakihara et al. | Sep 1998 | A |
| 5805462 | Poirot et al. | Sep 1998 | A |
| 5857204 | Lordi et al. | Jan 1999 | A |
| 5890997 | Roth | Apr 1999 | A |
| 5923832 | Shirakihara et al. | Jul 1999 | A |
| 5924095 | White | Jul 1999 | A |
| 5930794 | Linenbach et al. | Jul 1999 | A |
| 5933640 | Dion | Aug 1999 | A |
| 5966072 | Stanfill et al. | Oct 1999 | A |
| 5999729 | Tabloski, Jr. et al. | Dec 1999 | A |
| 6006242 | Poole et al. | Dec 1999 | A |
| 6012094 | Leyman | Jan 2000 | A |
| 6014670 | Zamanian et al. | Jan 2000 | A |
| 6016516 | Horikiri | Jan 2000 | A |
| 6032158 | Mukhopadhhyay et al. | Feb 2000 | A |
| 6038558 | Powers et al. | Mar 2000 | A |
| 6044211 | Jain | Mar 2000 | A |
| 6044374 | Nesamoney et al. | Mar 2000 | A |
| 6044394 | Cadden et al. | Mar 2000 | A |
| 6088716 | Stanfill et al. | Jul 2000 | A |
| 6145017 | Ghaffari | Nov 2000 | A |
| 6173276 | Kant et al. | Jan 2001 | B1 |
| 6208345 | Sheard et al. | Mar 2001 | B1 |
| 6256637 | Venkatesh et al. | Jul 2001 | B1 |
| 6259988 | Galkowski et al. | Jul 2001 | B1 |
| 6272650 | Meyer et al. | Aug 2001 | B1 |
| 6301601 | Helland | Oct 2001 | B1 |
| 6314114 | Coyle et al. | Nov 2001 | B1 |
| 6324437 | Frankel et al. | Nov 2001 | B1 |
| 6330008 | Razdow et al. | Dec 2001 | B1 |
| 6332212 | Organ et al. | Dec 2001 | B1 |
| 6339775 | Zamanian et al. | Jan 2002 | B1 |
| 6400996 | Hoffberg et al. | Jun 2002 | B1 |
| 6401216 | Meth et al. | Jun 2002 | B1 |
| 6437796 | Sowizral et al. | Aug 2002 | B2 |
| 6449711 | Week | Sep 2002 | B1 |
| 6480876 | Rehg et al. | Nov 2002 | B2 |
| 6496961 | Gupta et al. | Dec 2002 | B2 |
| 6538651 | Hayman et al. | Mar 2003 | B1 |
| 6584581 | Bay et al. | Jun 2003 | B1 |
| 6608628 | Ross et al. | Aug 2003 | B1 |
| 6611862 | Reisman | Aug 2003 | B2 |
| 6651234 | Gupta et al. | Nov 2003 | B2 |
| 6654907 | Stanfill et al. | Nov 2003 | B2 |
| 6658464 | Reisman | Dec 2003 | B2 |
| 6715145 | Bowman-Amuah | Mar 2004 | B1 |
| 6728879 | Atkinson | Apr 2004 | B1 |
| 6760903 | Morshed et al. | Jul 2004 | B1 |
| 6813761 | Das et al. | Nov 2004 | B1 |
| 6816825 | Ashar et al. | Nov 2004 | B1 |
| 6832369 | Kryka et al. | Dec 2004 | B1 |
| 6848100 | Wu et al. | Jan 2005 | B1 |
| 6879946 | Rong et al. | Apr 2005 | B2 |
| 6951013 | Lozins | Sep 2005 | B1 |
| 7062483 | Ferrari et al. | Jun 2006 | B2 |
| 7082604 | Schneiderman | Jul 2006 | B2 |
| 7085426 | August | Aug 2006 | B2 |
| 7103597 | McGovern | Sep 2006 | B2 |
| 7103620 | Kunz et al. | Sep 2006 | B2 |
| 7130484 | August | Oct 2006 | B2 |
| 7137116 | Parkes et al. | Nov 2006 | B2 |
| 7164422 | Wholey et al. | Jan 2007 | B1 |
| 7165030 | Yi et al. | Jan 2007 | B2 |
| 7167850 | Stanfill | Jan 2007 | B2 |
| 7316001 | Gold et al. | Jan 2008 | B2 |
| 7356819 | Ricart et al. | Apr 2008 | B1 |
| 7398514 | Ulrich et al. | Jul 2008 | B2 |
| 7417645 | Beda et al. | Aug 2008 | B2 |
| 7457984 | Kutan | Nov 2008 | B2 |
| 7467383 | Inchingolo et al. | Dec 2008 | B2 |
| 7505975 | Luo | Mar 2009 | B2 |
| 7577628 | Stanfill | Aug 2009 | B2 |
| 7636699 | Stanfill | Dec 2009 | B2 |
| 7716630 | Wholey et al. | May 2010 | B2 |
| 7840949 | Schumacher et al. | Nov 2010 | B2 |
| 7870556 | Wholey et al. | Jan 2011 | B2 |
| 7877350 | Stanfill et al. | Jan 2011 | B2 |
| 8566641 | Douros et al. | Oct 2013 | B2 |
| 20010055019 | Sowizral et al. | Dec 2001 | A1 |
| 20020080181 | Razdow et al. | Jun 2002 | A1 |
| 20020087921 | Rodriguez | Jul 2002 | A1 |
| 20020091747 | Rehg et al. | Jul 2002 | A1 |
| 20020091748 | Rehg et al. | Jul 2002 | A1 |
| 20020111876 | Rudraraju et al. | Aug 2002 | A1 |
| 20020129340 | Tuttle | Sep 2002 | A1 |
| 20020147745 | Houben et al. | Oct 2002 | A1 |
| 20020184616 | Chessell et al. | Dec 2002 | A1 |
| 20030004771 | Yaung | Jan 2003 | A1 |
| 20030023413 | Srinivasa | Jan 2003 | A1 |
| 20030033432 | Simpson et al. | Feb 2003 | A1 |
| 20030091055 | Craddock et al. | May 2003 | A1 |
| 20030126240 | Vosseler | Jul 2003 | A1 |
| 20030204804 | Petri et al. | Oct 2003 | A1 |
| 20040006745 | Van Heldan et al. | Jan 2004 | A1 |
| 20040041838 | Adusumilli et al. | Mar 2004 | A1 |
| 20040073529 | Stanfill | Apr 2004 | A1 |
| 20040093559 | Amaru et al. | May 2004 | A1 |
| 20040098452 | Brown et al. | May 2004 | A1 |
| 20040105391 | Charcranoon | Jun 2004 | A1 |
| 20040107414 | Bronicki et al. | Jun 2004 | A1 |
| 20040111469 | Manion et al. | Jun 2004 | A1 |
| 20040148373 | Childress et al. | Jul 2004 | A1 |
| 20040177099 | Ganesh et al. | Sep 2004 | A1 |
| 20040205726 | Chedgey et al. | Oct 2004 | A1 |
| 20040207665 | Mathur | Oct 2004 | A1 |
| 20040210831 | Feng et al. | Oct 2004 | A1 |
| 20040225657 | Sarkar | Nov 2004 | A1 |
| 20040260590 | Golani et al. | Dec 2004 | A1 |
| 20040268185 | Tan et al. | Dec 2004 | A1 |
| 20050021689 | Marvin et al. | Jan 2005 | A1 |
| 20050034112 | Stanfill | Feb 2005 | A1 |
| 20050039176 | Fournie | Feb 2005 | A1 |
| 20050059046 | Labrenz et al. | Mar 2005 | A1 |
| 20050086360 | Mamou et al. | Apr 2005 | A1 |
| 20050097515 | Ribling | May 2005 | A1 |
| 20050097561 | Schumacher et al. | May 2005 | A1 |
| 20050102670 | Bretl et al. | May 2005 | A1 |
| 20050144277 | Flurry et al. | Jun 2005 | A1 |
| 20050144596 | McCullough et al. | Jun 2005 | A1 |
| 20050149935 | Benedetti | Jul 2005 | A1 |
| 20050172268 | Kuturianu et al. | Aug 2005 | A1 |
| 20050177531 | Bracewell | Aug 2005 | A1 |
| 20050193056 | Schaefer et al. | Sep 2005 | A1 |
| 20050216421 | Barry et al. | Sep 2005 | A1 |
| 20050240621 | Robertson et al. | Oct 2005 | A1 |
| 20050262470 | Gavrilov | Nov 2005 | A1 |
| 20050289527 | Illowsky et al. | Dec 2005 | A1 |
| 20060085462 | Todd | Apr 2006 | A1 |
| 20060095722 | Biles et al. | May 2006 | A1 |
| 20060098017 | Tarditi et al. | May 2006 | A1 |
| 20060130041 | Pramanick et al. | Jun 2006 | A1 |
| 20060206872 | Krishnaswamy | Sep 2006 | A1 |
| 20060282474 | MacKinnon | Dec 2006 | A1 |
| 20060294150 | Stanfill et al. | Dec 2006 | A1 |
| 20060294459 | Davis et al. | Dec 2006 | A1 |
| 20070011668 | Wholey et al. | Jan 2007 | A1 |
| 20070022077 | Stanfill | Jan 2007 | A1 |
| 20070035543 | David et al. | Feb 2007 | A1 |
| 20070094211 | Sun et al. | Apr 2007 | A1 |
| 20070118839 | Berstis et al. | May 2007 | A1 |
| 20070139441 | Lucas et al. | Jun 2007 | A1 |
| 20070143360 | Harris et al. | Jun 2007 | A1 |
| 20070150429 | Huelsman et al. | Jun 2007 | A1 |
| 20070174185 | McGovern | Jul 2007 | A1 |
| 20070179923 | Stanfill | Aug 2007 | A1 |
| 20070239766 | Papaefstathiou et al. | Oct 2007 | A1 |
| 20070271381 | Wholey et al. | Nov 2007 | A1 |
| 20070285440 | MacInnis et al. | Dec 2007 | A1 |
| 20080049022 | Sherb et al. | Feb 2008 | A1 |
| 20080126755 | Wu et al. | May 2008 | A1 |
| 20080244524 | Kelso | Oct 2008 | A1 |
| 20080250049 | Chakra et al. | Oct 2008 | A1 |
| 20080288608 | Johnson | Nov 2008 | A1 |
| 20080294615 | Furuya et al. | Nov 2008 | A1 |
| 20090030863 | Stanfill et al. | Jan 2009 | A1 |
| 20090064147 | Beckerle et al. | Mar 2009 | A1 |
| 20090083313 | Stanfill et al. | Mar 2009 | A1 |
| 20090113196 | Jan et al. | Apr 2009 | A1 |
| 20090182728 | Anderson | Jul 2009 | A1 |
| 20090193391 | Miller et al. | Jul 2009 | A1 |
| 20090193417 | Kahlon | Jul 2009 | A1 |
| 20090224941 | Kansal et al. | Sep 2009 | A1 |
| 20090235267 | McKinney et al. | Sep 2009 | A1 |
| 20090245426 | Ratnakar et al. | Oct 2009 | A1 |
| 20090327196 | Studer et al. | Dec 2009 | A1 |
| 20100042976 | Hines | Feb 2010 | A1 |
| 20100070955 | Kahlon | Mar 2010 | A1 |
| 20100169137 | Jastrebski et al. | Jul 2010 | A1 |
| 20100174694 | Staebler et al. | Jul 2010 | A1 |
| 20100180344 | Malyshev et al. | Jul 2010 | A1 |
| 20100211953 | Wakeling et al. | Aug 2010 | A1 |
| 20100218031 | Agarwal et al. | Aug 2010 | A1 |
| 20100281462 | Festa | Nov 2010 | A1 |
| 20100281488 | Krishnamurthy et al. | Nov 2010 | A1 |
| 20110078500 | Douros et al. | Mar 2011 | A1 |
| 20110093433 | Stanfill et al. | Apr 2011 | A1 |
| 20110239186 | Zinkovsky et al. | Sep 2011 | A1 |
| 20110307897 | Atterbury et al. | Dec 2011 | A1 |
| 20120036498 | Akirekadu et al. | Feb 2012 | A1 |
| 20120054255 | Buxbaum et al. | Mar 2012 | A1 |
| 20120216176 | Gaikwad et al. | Aug 2012 | A1 |
| 20120233599 | Valdiviezo Basauri et al. | Sep 2012 | A1 |
| 20130290928 | Johnson | Oct 2013 | A1 |
| 20140026125 | Pasala et al. | Jan 2014 | A1 |
| 20140068566 | Coronado et al. | Mar 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 0834810 | Apr 1998 | EP |
| 64-013189 | Jan 1989 | JP |
| 06-236276 | Aug 1994 | JP |
| 08-278892 | Oct 1996 | JP |
| 08-305576 | Nov 1996 | JP |
| 63-231613 | Sep 1998 | JP |
| 11-184766 | Jul 1999 | JP |
| 2000-99317 | Apr 2000 | JP |
| 2006-504160 | Feb 2006 | JP |
| WO 9800791 | Jan 1998 | WO |
| WO 0211344 | Feb 2002 | WO |
| WO 2005001687 | Jan 2005 | WO |
| WO 2005086906 | Sep 2005 | WO |
| WO 2008124319 | Oct 2008 | WO |
| WO 2009039352 | Mar 2009 | WO |
| Entry |
|---|
| Mattson et al., “Patterns for Parallel Programming,” Addison-Wesley Professional ISBN: 0-321-22811-1 (2004). |
| European Examination Report issued in related EP Application No. 10 741 775.0 on Jul. 7, 2014. |
| International Search Report and Written Opinion, International Application No. PCT/US2013/076416, mailed Apr. 9, 2014 (10 pages). |
| Babaoglu, O et al., “Mapping parallel computations onto distributed systems in Paralex” Compuero '91. Advanced Computer Technology, Reliable Systems and Applications. 5th Annual European Computer Conference. Proceedings. Bologna, Italy May 13-16, 1991, Los Alamitos, CA, USA, IEEE Comput. Soc, US, May 13, 1991, pp. 123-130. |
| Baer, J.L. et al., “Legality and Other Properties of Graph Models of Computations.” Journal of the Association for Computing Machinery, vol. 17, No. 3, Jul. 1970, pp. 543-554. |
| Bookstein, A. et al., “Modeling Word Occurrences for the Compression of Concordances.” ACM Transactions on Information Systems, vol. 15, No. 3, Jul. 1997, pp. 254-290. |
| Burch, J.R. et al., “Sequential circuit verification using symbolic model checking.” In Design Automation Conference, 1990, Proceedings of the 27th ACM/IEEE. Jun. 24-28, 1990, pp. 46-51. |
| Control-M; New Dimension Software. User Manual. New Dimension Software Ltd., 1999. |
| Cytron, Ron et al., “Efficiently Computing Static Single Assignment Form and the Control Dependence Graph.” ACM Transactions on Programming Languages and Systems, vol. 13, No. 4, Oct. 1991, pp. 451-490. |
| Ebert, Jurgen et al., “A Declarative Approach to Graph-Based Modeling.” Workshop on Graph-Theoretic Concepts in Computer Science, 1994, pp. 1-19. |
| Gamma et al. “Design Patterns: Elements of Reusable Object-Oriented Software”, Sep. 1999. |
| Guyer et al., “Finding Your Cronies: Static Analysis for Dynamic Object Colocation.” Oct. 2004, ACM, pp. 237-250. |
| Grove et al., “A Framework for Call Graph Construction Algorithms.” Nov. 2001, ACM TOPLAS, vol. 23, Issue 6, pp. 685-746. |
| Herniter, Marc E., “Schematic Capture with MicroSim PSpice,” 2nd Edition, Prentice Hall, Upper Saddle River, N.J., 1996, pp. 51-52, 255-280, 292-297. |
| Jawadi, Ramamohanrao et al., “A Graph-based Transaction Model for Active Databases and its Parallel Implementation.” U. Florida Tech. Rep TR94-0003, 1994, pp. 1-29. |
| Just et al., “Review and Analysis of Synthetic Diversity for Breaking Monocultures.” Oct. 2004, ACM, pp. 23-32. |
| Kebschull, U. et al., “Efficient Graph-Based Computation and Manipulation of Functional Decision Diagrams.” University of Tubingen, 1993 IEEE, pp. 278-282. |
| Krahmer et al., “Graph-Based Generation of Referring Expressions.” Mar. 2003, MIT Press, vol. 29, No. 1, pp. 53-72. |
| Krsul, Ivan et al., “VMPlants: Providing and Managing Virtual Machine Execution Environments for Grid Computing.” Proceedings of the ACM/IEEE SC2004 Conference on Supercomputing, 2001, Nov. 6-12, 2004, 12 pages. |
| Li, Xiqing et al., “A Practical External Sort for Shared Disk MPPs.” Proceedings of Supercomputing '93, 1993, 24 pages. |
| Martin, David et al., “Models of Computations and Systems—Evaluation of Vertex Probabilities in Graph Models of Computations.” Journal of the Association for Computing Machinery, vol. 14, No. 2, Apr. 1967, pp. 281-299. |
| Ou, Chao-Wei et al., “Architecture-Independent Locality-Improving Transformations of Computational Graphs Embedded in k-Dimensions.” Proceedings of the 9th International Conference on Supercomputing, 1995, pp. 289-298. |
| “RASSP Data Flow Graph Design Application Note.” International Conference on Parallel Processing, Dec. 2000, Retrieved from Internet <http://www.atl.external.lmco.com/projects/rassp/RASSP—legacy/appnotes/FLOW/APNOTE—FLOW—02 >, 5 pages. |
| Romberg, M., “UNICORE: Beyond Web-based Job-Submission,” Proceedings of the 42nd Cray User Group Conference, Noordwijk (May 22-26, 2000). |
| Stanfill, Craig, “Massively Parallel Information Retrieval for Wide Area Information Servers.” 1991 IEEE International Conference on Systems, Man and Cybernetics, Oct. 1991, pp. 679-682. |
| Stanfill, Craig et al., “Parallel Free-Text Search on the Connection Machine System.” Communications of the ACM, vol. 29, No. 12, Dec. 1986, pp. 1229-1239. |
| Stanfill, Craig, “The Marriage of Parallel Computing and Information Retrieval.” IEE Colloquium on Parallel Techniques for Information Retrieval, Apr. 1989, 5 pages. |
| “Unicenter AutoSys Job Management,” Computer Associates, Copyright 2001. |
| Vajracharya, Suvas et al., “Asynchronous Resource Management.” Proceedings of the 15th International Parallel and Distributed Processing Symposium, Apr. 2001, 10 pages. |
| “Visual Lint: Squash Bugs Early with Interactive C/C++, C# and Java Code Analysis for Microsoft Visual Studio and Eclipse,” [ retrieved from the internet Dec. 3, 2012: www.riverblade.co.uk/products/visual—lint.] (2 pages). |
| Wah, B.W. et al., “Report on Workshop on High Performance Computing and Communications for Grand Challenge Applications: Computer Vision, Speech and Natural Language Processing, and Artificial Intelligence.” IEEE Transactions on Knowledge and Data Engineering, vol. 5, No. 1, Feb. 1993, 138-154. |
| “Topological sorting,” Wikipedia, accessed Dec. 10, 2012, 2 pages. |
| Russell, Nick, et al., “Workflow Control-Flow Patterns A Revised View,” Workflow Patterns Initiative, 2006, pp. 1-134. |
| van der Aalst, W.M.P., et al., “Workflow Patterns,” Distributed and Parallel Databases, 14, 5-51, 2003. |
| Shoten, Iwanami, “Encyclopedic Dictionary of Computer Science,” (with English Translation), May 25, 1990, p. 741. |
| Dillon, Laura K., et al., “Inference Graphs: A Computational Structure Supporting Generation of Customizable and Correct Analysis Components,” IEEE Transactions on Software Engineering, vol. 29, No. 2, Feb. 2003, pp. 133-150. |
| Evripidou, Paraskevas, et al., “Incorporating input/output operations into dynamic data-flow graphs,” Parallel Computing 21 (1995) 1285-1311. |
| Frankl, Phyllis G., et al., “An Applicable Family of Data Flow Testing Criteria,” IEEE Transactions on Sofrware Engineering, vol. 14, No. 10, Oct. 1988, pp. 1483-1498. |
| Whiting, Paul G., et al., “A History of Data-Flow Languages,” IEEE Annals of the History of Computing, vol. 16, No. 4, 1994, pp. 38-59. |
| International Search Report & Written Opinion received in PCT application No. PCT/US2011/040440, mailed Oct. 12, 2011, 13 pages. |
| European Search Report issued in application No. EP10003554, dated Sep. 24, 2010, 7 pages. |
| International Search Report & Written Opinion issued in PCT application No. PCT/US01/23552, mailed Jan. 24, 2002, 5 pages. |
| International Search Report & Written Opinion issued in PCT application No. PCT/US06/24957, dated Jan. 17, 2008, 14 pages. |
| International Search Report & Written Opinion issued in PCT application No. PCT/US07/75576, mailed Sep. 16, 2008, 13 pages. |
| International Search Report & Written Opinion issued in PCT application No. PCT/US08/71206, mailed Oct. 22, 2008, 12 pages. |
| International Search Report & Written Opinion issued in PCT application No. PCT/US10/49966, dated Nov. 23, 2010, 8 pages. |
| International Search Report & Written Opinion received in PCT application No. PCT/US10/24036, mailed Mar. 23, 2010, 11 pages. |
| Supplemental European Search Report issued in application No. EP07813940, dated Nov. 26, 2009, 7 pages. |
| Supplemental European Search Report issued in application No. EP08796632, dated Sep. 24, 2010, 6 pages. |
| Japanese Office Action, with English Translation, JP application No. 2011-000948, mailed Jan. 8, 2013, 11 pages. |
| Supplemental European Search Report issued in application No. EP06774092, dated Dec. 19, 2012, 5 pages. |
| European Search Report issued in application No. EP10741775, dated Nov. 14, 2012, 4 pages. |
| Japanese Office Action, with English Translation, JP application No. 2009-523997, mailed Oct. 23, 2012, 7 pages. |
| Japanese Office Action, with English Translation, JP application No. 2008-519474, mailed Sep. 25, 2012, 8 pages. |
| Japanese Office Action, with English Translation, JP application No. 2010-518415, mailed Feb. 21, 2013, 11 pages. |
| Extended European Search Report, EP 12165575, mailed May 10, 2013, 9 pages. |
| U.S. Appl. No. 09/627,252, filed Jul. 28, 2000. |
| U.S. Appl. No. 10/268,509, filed Oct. 10, 2002. |
| U.S. Appl. No. 11/467,724, filed Aug. 28, 2006. |
| U.S. Appl. No. 11/733,579, filed Apr. 10, 2007. |
| U.S. Appl. No. 11/169,014, filed Jun. 27, 2005. |
| U.S. Appl. No. 11/167,902, filed Jun. 27, 2005. |
| U.S. Appl. No. 12/977,545, filed Dec. 23, 2010. |
| U.S. Appl. No. 11/836,349, filed Aug. 9, 2007. |
| U.S. Appl. No. 12/180,141, filed Jul. 25, 2008. |
| U.S. Appl. No. 12/704,998, filed Feb. 12, 2010. |
| U.S. Appl. No. 13/161,010, filed Jun. 15, 2011. |
| U.S. Appl. No. 12/638,588, filed Dec. 15, 2009. |
| U.S. Appl. No. 13/678,921, filed Nov. 16, 2012. |
| U.S. Appl. No. 13/678,928, filed Nov. 16, 2012. |
| U.S. Appl. No. 13/733,403, filed Jan. 3, 2013. |
| Number | Date | Country | |
|---|---|---|---|
| 20140189653 A1 | Jul 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13733403 | Jan 2013 | US |
| Child | 14165698 | US |
