SYSTEM AND METHOD FOR TESTING COMPILER IMPLEMENTATION USING DYNAMIC COMBINATORIAL TEST GENERATION

Abstract

A system and method can support compiler testing in a programming language environment. The programming language environment can include a testing framework that can provide a program template based on a programming language that is targeted by a compiler implementation to be tested. Furthermore, the testing framework can generate a plurality of sub-tests that contains different variations in the test space based on the program template, and can use the compiler implementation to be tested to compile the generated sub-tests. Finally, the testing framework can check the output of the compiler implementation to be tested against the expected output for the generated sub-tests.

Description

FIELD OF INVENTION

The present invention is generally related to computer systems and software, and is particularly related to supporting compiler testing in a programming language environment.

BACKGROUND

Programming language can be used to write software source code, or source file, that contains various instructions. The source code, or source file, can be compiled into executables that can run on different types of computers, or computational devices. As the programming language becomes more complex, the risk due to regression and interaction between various programming language features can become more serious. The traditional compiler testing can become inadequate to verify whether the various modifications or enhancements to the programming language compiler work appropriately. For example, existing methodologies for writing compiler tests can be insufficient to provide adequate test coverage. This is because that the test space in compiler implementations can be enormous, and a single test program may only be able to cover a small fraction of the whole test space.

This is the general area that embodiments of the invention are intended to address.

SUMMARY

Described herein are systems and methods that can support compiler testing in a programming language environment. The programming language environment can include a testing framework that can provide a program template based on a programming language that is targeted by a compiler implementation to be tested. Furthermore, the testing framework can generate a plurality of sub-tests that contain different variations in the test space based on the program template, and can use the compiler implementation to be tested to compile the generated sub-tests. Finally, the testing framework can check the output of the compiler implementation to be tested against the expected output for the generated sub-tests.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an illustration of supporting dynamic combinatorial test generation in a programming language environment in accordance with various embodiments of the invention.

FIG. 2 illustrates an exemplary flow chart for supporting dynamic combinatorial test generation in a programming language environment in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

Described herein are systems and methods that can support compiler testing in a programming language environment.

In accordance with an embodiment of the invention, compiler tests can be developed for testing different compiler implementations. The compiler test can be implemented using a programming language accepted by the compiler. The compiler test can verify whether programs that are considered to be valid program by the programming language specification are accepted by the compiler implementation under test. On the other hand, the compiler test can also check whether invalid programs are correctly rejected by the compiler implementation.

There can be two types of tests: positive tests, which use test programs that are expected to be accepted by the compiler, and negative tests, which use test programs that are expected to be rejected by the compiler.

The compiler tests can be developed in hand-written standalone tests. This approach can become inadequate for testing modern compilers, since the compiler feature can span across multiple dimensions in the test space. A simple standalone test may only represent a single point in a large test space. As a result, the percentage of the test space covered by the hand-written standalone tests can be insufficient.

For example, in a programming language such as JAVA, which supports arithmetic operations on different numeric types (e.g. int, long, double, float, short, and byte), the testing of a compiler implementation on whether the simple program “1+1” is accepted may require 36 combinations. In addition, the compiler test may need to take into consideration the type conversions (such as widening an int result to a long result), which may increase the number of combinations required to 216.

Thus, only relying on hand-written standalone tests might be impractical to stress test a compiler implementation. For a simple test, as a trade-off, the test developer may chose to write standalone tests for a few selected cases. On the other hand, the test program may need to handle millions of combinations in the cases of more complicated interactions between various programming language features.

In accordance with an embodiment of the invention, the compiler tests can dynamically generate different sub-tests embodying interesting variations for a given program pattern. For example, a single test program including a few hundred lines can generate and test thousands or millions of useful combinations. Thus, the generated sub-tests can be beneficial in finding errors in the compiler implementation.

Dynamic Combinatorial Test Generation

FIG. 1 shows an illustration of supporting dynamic combinatorial test generation in a programming language environment in accordance with various embodiments of the invention. As shown in FIG. 1, a programming language environment 100 can use a testing framework 101 to test a compiler implementation 102. The compiler implementation 102 can target a programming language 110 in the programming language environment 100.

The testing framework 101 can use a program template 103, which can be based on the programming language 110 that is targeted by the compiler implementation 102. Furthermore, the testing framework 101 can generate a plurality of sub-tests, e.g. sub-test A-B 111-112, that represents different variations based on the program template 103. Then, the compiler implementation 102 can be used to compile the sub-tests A-B 111-112, and produce corresponding compilation outputs A-B 121-122.

Furthermore, the program template 103 can include one or more place-holders 104 that can capture various features specified in the programming language 110. The testing framework 101 can use the different place-holders 104 to inject actual code-snippets 105 into the program template 103 for generating different sub-test A-B 111-112. Here, the code-snippets 105 can be implemented using the programming language 110. Additionally, the generation of the code-snippets 105 can be in a recursive fashion, e.g. some code-snippets 105 can be generated using other templates.

The testing framework 101 allows the different variations in the sub-tests A-B 111-112 to be generated combinatorially based on the program template 103, by varying relevant programming language features, or relevant dimensions of choice in the test space. For example, in an object-oriented programming language such as JAVA, the relevant programming language features can include class or method accessibility (such as public, private, or protected), program structure elements (such as class, interface, abstract class), types of variables (such as int, long, String), and method signature (such as the number of arguments and whether or not a method throws exceptions, etc.)

On the other hand, the testing framework 101 can avoid generating sub-tests for loose-coupled, or irrelevant, programming language features, e.g. the combination of an addition operation with a printing feature.

In accordance with various embodiments of the invention, the number of the generated sub-tests represents the number of variations in the test space based on the program template 103. It can be beneficial to use the dynamic combinatorial test generation approach to generate a large number of sub-tests. Also, the testing framework 101 can discard one or more generated sub-tests, which are considered less important, when the total number of generated sub-tests exceeds a maximum number.

Additionally, the testing framework 101 can compute an expected output for each generated sub-test A-B 111-112. Then, the testing framework 101 can compare the actual output of the compiler implementation, e.g. compilation output A-B 121-122, against the expected output for each sub-test, e.g. expected output A-B 131-132. Furthermore, the strength of a compiler test may depend on how precise the expected output is.

For example, the testing framework 101 can generate various sub-tests for testing whether a compiler implementation can handle the arithmetic operations on different numeric types correctly. One sub-test can include the source code that adds an integer, e.g. “1”, to a String, e.g. “banana.” The compiler implementation is expected to produce an error message indicating that the operation is inappropriate. A weak test can simply verify that the compilation fails, while a stronger test can verify that the compiler implementation 102 can provide a correct error message.

On the other hand, another sub-test can be generated to include the source code that adds an integer, e.g. “1”, to a float, e.g. “1.0”. In this case, the compiler implementation is expected to perform a type conversion and create executables that can be run on a computer and can produce a result of a float, e.g. “2.0”.

An Exemplary Testing Framework

In accordance with various embodiments of the invention, a combo-test can be developed based on a testing framework. Attached is Appendix A that provides information regarding an exemplary testing framework. The information in Appendix A is provided for illustration purposes and should not be construed to limit all of the embodiments of the invention.

In Appendix A, a combo-test, i.e. an instance of the LambdaConversionTest Class, can be used to test a JAVA compiler implementation. The LambdaConversionTest can extend from a ComboTestBase base class, or interface (Line 1). The LambdaConversionTest can define seven dimension variables using JAVA annotations, e.g. @DimensionVar. Furthermore, each sub-test generated by the LambdaConversionTest can include three JAVA source files, as defined using JAVA annotations, e.g. @SourceFile.

The LambdaConversionTest can include a set of source templates written in JAVA with each source template associated with a different source file. The source templates can contain one or more place-holders, each of which can be used to inject custom code-snippets into the source template. A sub-test, which can be considered as an equivalent of a single standalone test, is created when the LambdaConversionTest finishes replacing the place-holders in the set of source templates with the actual code-snippets.

The source template can be defined in the format of either a string or a file. For example, the source template for creating a JAVA source file, Sam.java, includes a string of samSource, which is defined as “#{PACKAGE.PACKAGE_DECL}\n#{CLASS}”. The LambdaConversionTest can use the string of samSource to generate different Sam.java JAVA source files in different sub-tests, by injecting different code-snippets that defines the values of PackageKind and SamKind in samSource. The different placeholders in the string of samSource can be used to vary different dimension variables in the test space, e.g. PACKAGE and CLASS. Thus, the LambdaConversionTest can produce different variations for the same test pattern.

Furthermore, each relevant dimensional variable can be associated with a particular range of variations or alternatives. A testing framework can iterate over the relevant dimensional variables in the test space as shown in the following.

public static void main(String[ ] args) throws Exception {
for (PackageKind samPkg : PackageKind.values( )) {
for (ModifierKind modKind : ModifierKind.values( )) {
for (SamKind samKind : SamKind.values( )) {
for (MethodKind meth : MethodKind.values( )) {
for (TypeKind retType : TypeKind.values( )) {
for (TypeKind argType : TypeKind.values( )) {
for (TypeKind thrownType : TypeKind.values( )) {
new LambdaConversionTest(samPkg, modKind,
samKind,
meth, retType, argType, thrownType).test( );
}
}
}
}
}
}
}
}

Additionally, the combo-test can compute the expected output for each generated sub-test, which can be expressed as a function of the dimensions of the test-space covered by the test. The process can be repeated by generating all the relevant combinations of the dimension variables, replacing the source template place-holders with different snippets, thus achieving slightly different sub-tests/expected output pairs for each generated sub-test.

Once the sub-tests/expected output pairs have been generated, each generated sub-test can be compiled and executed separately, and the output from the compiler implementation can be checked against the expected output associated with each sub-test. For example, the LambdaConversionTest includes a postCompile method (Lines 25-40) that not only can verify whether a compilation is succeed or failed as expected, but also can verify whether the compilation is has failed for a right reason.

Thus, the testing framework can verify the handling of subtle interactions between relevant programming language features in a compiler implementation, and can provide a high level of confidence that a feature in a compiler implementation has been implemented completely and correctly.

FIG. 2 illustrates an exemplary flow chart for supporting dynamic combinatorial test generation in a programming language environment in accordance with an embodiment of the invention. As shown in FIG. 2, at step 201, a testing framework can provide a program template based on a programming language that is targeted by a compiler implementation to be tested. Then, at step 202, the testing framework can generate a plurality of sub-tests that are based on different variations of the program template. Furthermore, at step 203, the testing framework can compile the generated sub-tests using the compiler implementation to be tested. Finally, at step 204, the testing framework can check the output of the compiler implementation against the expected output associated with the sub-tests.

The present invention may be conveniently implemented using one or more conventional general purpose or specialized digital computer, computing device, machine, or microprocessor, including one or more processors, memory and/or computer readable storage media programmed according to the teachings of the present disclosure. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art.

In some embodiments, the present invention includes a computer program product which is a storage medium or computer readable medium (media) having instructions stored thereon/in which can be used to program a computer to perform any of the processes of the present invention. The storage medium can include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.

The foregoing description of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to the practitioner skilled in the art. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalence.

Appendix A

The following provides information regarding an exemplary testing framework. The information is provided for illustration purposes and should not be construed to limit all of the embodiments of the invention.

An example of a combo-test
1   public class LambdaConversionTest extends
    ComboTestBase<LambdaConversionTest> {
2
3   @Factory
4   public static Object[ ] testCombo( ) throws Exception {
5   return factory(LambdaConversionTest.class);
6   }
7
8   @DimensionVar(“PACKAGE”) PackageKind samPkg;
9   @DimensionVar(“MODIFIER”) ModifierKind modKind;
10  @DimensionVar(“CLASS”) SamKind samKind;
11  @DimensionVar(“METH”) MethodKind meth;
12  @DimensionVar(“RET”) TypeKind retType;
13  @DimensionVar(“ARG”) TypeKind argType;
14  @DimensionVar(“THROWN”) TypeKind thrownType;
15
16  @SourceFile(“Sam.java”)
17  String samSource = “#{PACKAGE.PACKAGE_DECL} \n
    #{CLASS}”;
18
19  @SourceFile(“PackageClass.java”)
20  String packageSource = “#{PACKAGE.PACKAGE_DECL}\n
    #{MODIFIER} class PackageClass extends Exception { }”;
21
22  @SourceFile(“Client.java”)
23  String clientSource =
    “#{PACKAGE.IMPORT_STATEMENT}\n
    class Client { Sam s = x −> null; }”;
24
25  @Override
26  protected void postCompile(String group) {
27  if (samKind != SamKind.INTERFACE) {
28  assertCompileFailed(“SAM type must be an interface”);
29  } else if (meth != MethodKind.NON_GENERIC) {
30  assertCompileFailed(“target method must be non-generic”);
31  } else if (samPkg != PackageKind.NO_PKG &&
32  modKind != ModifierKind.PUBLIC &&
33  (retType == TypeKind.PKG_CLASS
34  | | argType == TypeKind.PKG_CLASS
35  | | thrownType ==
    TypeKind.PKG_CLASS)) {
36  assertCompileFailed(“target must not contain
    inaccessible types”);
37  } else {
38  assertCompileSucceeded( );
39  }
40  }
41
42  static enum PackageKind implements Template {
43  NO_PKG(“”),
44  PKG_A(“a”);
45
46  String pkg;
47
48  PackageKind(String pkg) {
49  this.pkg = pkg;
50  }
51
52  public String expand(String selector) {
53  if (this == NO_PKG)
54  return “”;
55  switch (selector) {
56  case “PACKAGE_DECL”: return
    String.format(“package %s;”, pkg);
57  case “IMPORT_STATEMENT”: return
    String.format(“import %s.*;”, pkg);
58  default: throw new IllegalArgumentException(selector);
59  }
60  }
61  }
62
63  static enum SamKind implements Template {
64  CLASS(“public class Sam { }”),
65  ABSTRACT_CLASS(“public abstract class Sam { }”),
66  ANNOTATION(“public @interface Sam { }”),
67  ENUM(“public enum Sam { }”),
68  INTERFACE(“public interface Sam { \n #{METH}; \n }”);
69
70  String template;
71
72  SamKind(String template) { this.template = template; }
73
74  public String expand(String selector) { return template; }
75  }
76
77  static enum ModifierKind implements Template {
78  PUBLIC(“public”),
79  PACKAGE(“”);
80
81  String template;
82
83  ModifierKind(String template) { this.template = template; }
84
85  public String expand(String selector) { return template; }
86  }
87
88  static enum TypeKind implements Template {
89  EXCEPTION(“Exception”),
90  PKG_CLASS(“PackageClass”);
91
92  String template;
93
94  private TypeKind(String template) { this.template =
    template; }
95
96  public String expand(String selector) { return template; }
97  }
98
99  static enum MethodKind implements Template {
100 NONE(“”),
101 NON_GENERIC(“public #{RET} m(#{ARG} s) throws
    #{THROWN};”),
102 GENERIC(“public <X> #{RET} m(#{ARG} s) throws
    #{THROWN};”);
103
104 String template;
105
106 private MethodKind(String template) { this.template =
    template; }
107
108 public String expand(String selector) { return template; }
109 }
110
111 }
112

Claims

1. A method for supporting compiler testing in a programming language environment, comprising: providing a program template based on a programming language that is targeted by a compiler implementation to be tested;generating a plurality of sub-tests that are based on different variations of the program template; andcompiling the generated sub-tests using the compiler implementation to be tested.

2. The method according to claim 1, further comprising: providing one or more place-holders in the program template, wherein the one or more place-holders are used to inject code-snippets in the program template.

3. The method according to claim 2, further comprising: allowing at least one code-snippet to be generated by templates.

4. The method according to claim 1, further comprising: computing an expected output associated with at least one sub-test in the plurality of sub-tests.

5. The method according to claim 4, further comprising: checking an output of the compiler implementation against the expected output associated with the at least one sub-test.

6. The method according to claim 1, further comprising: allowing the different variations of the program template to be generated combinatorially by varying relevant dimensions of choice in the program template.

7. The method according to claim 1, further comprising: providing a testing framework that operates to define a pluralilty of dimensional variables in a test space.

8. The method according to claim 7, further comprising: iterating over each said dimensional variables in the test space, wherein each dimensional variable is associated with a particular range of variations in the resulting sub-test.

9. The method according to claim 7, further comprising: allowing the testing framework to contain a plurality of source templates, wherein each source template is associate with a source file.

10. The method according to claim 1, further comprising: discarding one or more generated sub-tests when the generated sub-tests exceeds a maximum number.

11. A system for supporting compiler testing in a programming language environment, comprising: one or more microprocessors;a program template based on a programming language that is targeted by a compiler implementation to be tested; anda testing framework, running on the one or more microprocessors, that operates to perform the steps of: generating a plurality of sub-tests that are based on different variations of the program template; andcompiling the generated sub-tests using the compiler implementation to be tested.

12. The system according to claim 11, wherein: the program template contains one or more place-holders, wherein the one or more place-holders are used to inject code-snippets in the program template.

13. The system according to claim 12, wherein: at least one code-snippet is generated by templates.

14. The system according to claim 11, wherein: the testing framework operates to compute an expected output associated with at least one sub-test in the plurality of sub-tests.

15. The system according to claim 14, wherein: the testing framework operates to check an output of the compiler implementation against the expected output associated with the at least one sub-test.

16. The system according to claim 11, wherein: the testing framework operates to generate the different variations of the program template combinatorially by varying relevant dimensions of choice in the program template.

17. The system according to claim 11, wherein: the testing framework operates to define a pluralilty of dimensional variables in a test space.

18. The system according to claim 17, wherein: the testing framework operates to perform the further steps of: iterating over each said dimensional variables in the test space, wherein each dimensional variable is associated with a particular range of variations in the resulting sub-test; andallowing the testing framework to contain a plurality of source templates, wherein each source template is associate with a source file.

19. The system according to claim 11, wherein: the testing framework operates to discard one or more generated sub-tests when the generated sub-tests exceeds a maximum number.

20. A non-transitory machine readable storage medium having instructions stored thereon that when executed cause a system to perform the steps comprising: providing a program template based on a programming language that is targeted by a compiler implementation to be tested;generating a plurality of sub-tests that are based on different variations of the program template; andcompiling the generated sub-tests using the compiler implementation to be tested.