Patent Application
20140359579
Testing generally pertains to determining whether a system, such as a software application, operates correctly. More specifically, testing is a process of locating and reporting defects (a.k.a. bugs) and ensuring the defects are fixed by generating and executing tests on a system under test. In other words, testing is a cyclic workflow including running generated tests and fixing defects. As a result, testing can improve the quality and reliability of a system.
A test comprises a set of conditions under which a determination can be made as to whether a system is operating correctly with respect to a system requirement. An instance of a test, or test case, is characterized by a known input and an expected output determined prior to test execution. For example, data acquired from an external data source (e.g., database, XML document . . . ) can be input to a system under test, and a result produced in response to the input can be compared with an expected output to determine whether the system passed the test.
The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed subject matter. This summary is not an extensive overview. It is not intended to identify key/critical elements or to delineate the scope of the claimed subject matter. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
Briefly described, the subject disclosure pertains to tests that combine data and instructions. Data and instructions can be collocated and encoded within a programming-language source file. Data can be delimited within a programming language by at least one character denoting non-executing code. In one embodiment, the data can be encoded within a comment and distinctly annotated therein. Instructions can be specified to operate on the data in the programming language syntax. Further, various metadata can be included within one or both of a data portion and an instruction portion to facilitate specification of instructions that interact with data, among other things. A test file that combines data and instructions can be pre-processed to at least extract data and compile the instructions. During testing, the data can be loaded with respect to a system under test and the instructions executed on the data in an attempt to verify correct operation of the system under test.
To the accomplishment of the foregoing and related ends, certain illustrative aspects of the claimed subject matter are described herein in connection with the following description and the annexed drawings. These aspects are indicative of various ways in which the subject matter may be practiced, all of which are intended to be within the scope of the claimed subject matter. Other advantages and novel features may become apparent from the following detailed description when considered in conjunction with the drawings.
Software tests can be composed of data, instructions, or a mix of data and instructions. For example, a simple baseline test can involve sending input data to a system under test and comparing its output with the expected output. As another example, actions can be performed that correspond to one or more instructions, and actual results compared with expected results. While data-only and instruction-only test formats are trivial to design and implement, mixing data and instructions presents unique challenges, such as how to separate data and instructions, how to refer to areas of data when issuing instructions, and how to keep the two types of inputs comingled, among other things.
Previous efforts to address challenges of mixing data and instructions have been awkward at best. Some use a data-only format such as XML (eXtensible Markup Language) to encode instructions, thereby limiting the allowed complexity of the instruction set. Others have used an instruction-only format such as a general-purpose programming language to encode data. However, general-purpose programming languages are conventionally ill suited to working with large amounts of data, for instance where data is injected as a string into a local variable and operations are performed on the string. Finally, some tests use a separate data file and a separate instruction file, making it difficult to understand which instructions correspond to which data, complicating maintenance and comprehension.
Details below generally pertain to combined data and instruction test content. Data and instructions can be collocated, for example in a single file. Furthermore, a programming language can be employed wherein instructions are encoded in the programming language syntax and data portions are delimited by a particular character or sequence of characters denoting non-executing code. In one embodiment, data can be encoded within one or more comments. Further, the data can be distinctly annotated to enable data to be distinguished from conventional comments or like constructs. The programming language instructions can be specified to operate on the data. Additionally, metadata can be injected with respect one or both of data and instructions to at least aid specification of instructions that interact with data. A test that mixes data and instructions can be subject to pre-processing to acquire data and compile instructions, among other things. Subsequently, data can be loaded with respect to a system under test and instructions executed on the data in an attempt to verify correct operation of the system under test.
Various aspects of the subject disclosure are now described in more detail with reference to the annexed drawings, wherein like numerals refer to like or corresponding elements throughout. It should be understood, however, that the drawings and detailed description relating thereto are not intended to limit the claimed subject matter to the particular form disclosed. Rather, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claimed subject matter.
Referring initially to
The test file 130 can include data component 132 and instruction component 134. The instruction component 134 can comprise one or more source code instructions encoding test functionality. The data component 132 encodes data that can be operated on by one or more instructions in the instruction component 134. Further, the data can be embedded within one or more comments or like programming language construct that is not subject to execution and is typically ignored by compilers and interpreters. Further yet, data can be distinctly annotated with at least one predetermined character. In this manner, a data portion can be distinguished from conventional comments employed to make code easier to understand or like constructs. In one non-limiting example, where a conventional line comment is denoted with two forward slashes “//,” two additional forward slashes can be added resulting in four forward slashes “////” to distinguish data from conventional comments.
By collocating data and instructions that operate on the data, the test authoring and readability experiences is simplified as compared to conventional techniques, which necessitate authoring and maintenance of separate files for data and instructions or where data and instructions are not separately present (e.g., data injected into instructions). Furthermore, by encoding a test in programming language source code, the flexibility and functionality afforded by programming language can be exploited for testing.
Metadata can be added to one or more of the data component 132 and the instruction component 134 for a number of purposes. Generally, the metadata can refer to data about data and/or instructions. In one instance, the metadata can correspond to a marker. A marker is an anonymous or named element that identifies a particular location in data. The marker can be specified with respect to data and one or more instructions. In this manner, an instruction can specify a marker to go to, which is a location specified with a corresponding marker name in data. Moreover, a marker is not strictly positional as in designating specific line and position numbers but rather provided a simplified way to track and interact with data locations in a manner analogous to a bookmark. Metadata can also correspond to a file name thereby enabling multiple collections of data to be defined by name and employed by one or more instructions. Additionally, metadata can also correspond to a bug comment, which indicates a known bug, or defect, with respect to test. Consequently, even if the test passes, the result will be failure due to the identified defect. Still further, metadata can correspond to various options regarding the test such as, but not limited to, a compilation target.
Several examples of a test files including a combination of data and instructions and associated with particular test cases are provided below. These are simple examples presented to solely to aid clarity and understanding of various disclosed aspects. Accordingly, the examples are not intended to limit the scope of the claimed subject matter. Furthermore, the examples pertain to a testing programming language or more specifically a programming language service provided by an editor or an integrated development environment. However, this is just one of a plurality of domains to which the aspects are applicable.
Consider the following first example of test file content:
The first three lines include four forward slashes defining the data portion of this test, wherein the first two forward slashes indicate a convention line comment and the additional two forward slashes indicate that the comment corresponds to test data. Since this test is designed to test a language service associated with an editor, the data is embodied as code that a user would enter within an editor. The data also includes a marker encoded as a comment within a comment line (e.g., /*1*/) which declares a location in the data, or here code, where testing can be employed. The last two lines are instructions that define imperatively the text functionality. Here, a parameter help feature is to be tested in the case where the type of operand of the method being invoked was inferred. More particularly, since the type of “x” is string, the type of “y” should be “string,” and thus the type of the first argument to “substr” (a method defined on the string type) should be “number.” The first instruction “goTo.marker(“1”)” places the cursor, or caret, where maker “1” is located. In response, the language service can provide a pop-up box or window that identifies a parameter type. The second instruction seeks to verify that the parameter help type provided is “number.” If the type is “number,” the language service passed the test. Otherwise, the language service failed the test.
The next example seeks to test whether the correct member is supplied by a language service.
Similar to the first example, the first three lines include data embedded in comment lines as denoted by the four forward slashes. Further, the data portion includes an anonymous marker specified by the syntax of a forward slash followed by two stars and another forward slash (/**/). The last three lines include instructions and a bug comment. The first instruction moves the cursor or caret to the location following the period after “n.” The second instruction verifies there is not a member “C” included in a member list. A populated member list would be displayed upon positioning the cursor following the period at which point the list can be analyzed to determine if “C” is in the list. The test is failed if “C” is in the list, and otherwise, the test is passed. Here, however, the bug comment indicates that there is a defect with the test. Thus, regardless of the output of the test, the test will fail because of the bug comment.
The final example specifies a navigation test to validate a “Go to Definition” feature. Consider the following test file code:
In this example, there are two blocks of data defined by in a comment followed by “@Filename:,” namely “Definition.ts” and “Consumption.ts.” There are also three instructions that operator over the data. The first instruction moves the cursor to the first marker between “new” and “c( )” in the “Consumption.ts” data block. The second instruction navigates to the first block of data, namely “Definition.ts.” The third instruction seeks to verify that the cursor, or caret, is positioned at second marker. Subsequently the test can be passed or failed appropriately. In general, this test illustrates the use of multiple files and multiple markers to validate a specific language service feature.
Turning attention to
The parser component 310 is configured to identify and capture data encoded within a programming-language source file by way of lexical, syntactic, and sematic analysis associated with parsing. For example, upon identification of comment line syntax and one or more predetermined characters that indicate that data follows, the remainder of line can be captured as data and saved to another file. In one instance, the data can be removed from the file. However, the data need not be removed especially in the case where the data is encoded within one or more comments that are non-executable. In this instance, the data can simply be copied.
The metadata component 320 can operate in conjunction with the parser component to 310 to identify and remove or replace metadata present in a data portion or instruction portion. For example, metadata corresponding to a marker can be removed from the data portion and replaced in the instruction portion with an explicit position (e.g., line 2, column 4). Similarly, reference to code blocks from the instruction portion can be replaced with positional information with respect to the data. In another instance, a parser can identify metadata corresponding to a bug comment. Here, the metadata component 320 can be configured to add or modify an instruction to ensure that the test is not denoted as passed since, there is a known bug associated with the test.
The compiler component 330 is configured to compile instructions into executable instructions targeting a particular machine or execution environment (e.g., virtual machine . . . ). Further, compiled instructions can be stored alongside data. In other words, a file comprising the compiled instructions and a file comprising the data can be local to each other, for example within the container (e.g., folder) on a machine.
The aforementioned systems, architectures, environments, and the like have been described with respect to interaction between several components. It should be appreciated that such systems and components can include those components or sub-components specified therein, some of the specified components or sub-components, and/or additional components. Sub-components could also be implemented as components communicatively coupled to other components rather than included within parent components. Further yet, one or more components and/or sub-components may be combined into a single component to provide aggregate functionality. Communication between systems, components and/or sub-components can be accomplished in accordance with either a push and/or pull model. The components may also interact with one or more other components not specifically described herein for the sake of brevity, but known by those of skill in the art.
Furthermore, various portions of the disclosed systems above and methods below can include or employ of artificial intelligence, machine learning, or knowledge or rule-based components, sub-components, processes, means, methodologies, or mechanisms (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, classifiers . . . ). Such components, inter alia, can automate certain mechanisms or processes performed thereby to make portions of the systems and methods more adaptive as well as efficient and intelligent. By way of example, and not limitation, the pre-process component 210 can employ such mechanisms in conjunction with identification of data in file comprising a combination of data and instructions. For instance, such mechanisms can be utilized to identify data specified as comments without any distinguishing annotation.
In view of the exemplary systems described above, methodologies that may be implemented in accordance with the disclosed subject matter will be better appreciated with reference to the flow charts of
Referring to
The word “exemplary” or various forms thereof are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Furthermore, examples are provided solely for purposes of clarity and understanding and are not meant to limit or restrict the claimed subject matter or relevant portions of this disclosure in any manner. It is to be appreciated a myriad of additional or alternate examples of varying scope could have been presented, but have been omitted for purposes of brevity.
As used herein, the terms “component” and “system,” as well as various forms thereof (e.g., components, systems, sub-systems . . . ) are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an instance, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
The conjunction “or” as used in this description and appended claims is intended to mean an inclusive “or” rather than an exclusive “or,” unless otherwise specified or clear from context. In other words, “‘X’ or ‘Y’” is intended to mean any inclusive permutations of “X” and “Y.” For example, if “‘A’ employs ‘X,’” “‘A employs ‘Y,’” or “‘A’ employs both ‘X’ and ‘Y,’” then “‘A’ employs ‘X’ or ‘Y’” is satisfied under any of the foregoing instances.
Furthermore, to the extent that the terms “includes,” “contains,” “has,” “having” or variations in form thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.
In order to provide a context for the claimed subject matter,
While the above disclosed system and methods can be described in the general context of computer-executable instructions of a program that runs on one or more computers, those skilled in the art will recognize that aspects can also be implemented in combination with other program modules or the like. Generally, program modules include routines, programs, components, data structures, among other things that perform particular tasks and/or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the above systems and methods can be practiced with various computer system configurations, including single-processor, multi-processor or multi-core processor computer systems, mini-computing devices, mainframe computers, as well as personal computers, hand-held computing devices (e.g., personal digital assistant (PDA), phone, watch . . . ), microprocessor-based or programmable consumer or industrial electronics, and the like. Aspects can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all aspects of the claimed subject matter can be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in one or both of local and remote memory storage devices.
With reference to
The processor(s) 820 can be implemented with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any processor, controller, microcontroller, or state machine. The processor(s) 820 may also be implemented as a combination of computing devices, for example a combination of a DSP and a microprocessor, a plurality of microprocessors, multi-core processors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The computer 810 can include or otherwise interact with a variety of computer-readable media to facilitate control of the computer 810 to implement one or more aspects of the claimed subject matter. The computer-readable media can be any available media that can be accessed by the computer 810 and includes volatile and nonvolatile media, and removable and non-removable media. Computer-readable media can comprise computer storage media and communication media.
Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes memory devices (e.g., random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM) . . . ), magnetic storage devices (e.g., hard disk, floppy disk, cassettes, tape . . . ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ), and solid state devices (e.g., solid state drive (SSD), flash memory drive (e.g., card, stick, key drive . . . ) . . . ), or any other like mediums that can be used to store, as opposed to transmit, the desired information accessible by the computer 810. Accordingly, computer storage media excludes modulated data signals.
Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
Memory 830 and mass storage 850 are examples of computer-readable storage media. Depending on the exact configuration and type of computing device, memory 830 may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory . . . ) or some combination of the two. By way of example, the basic input/output system (BIOS), including basic routines to transfer information between elements within the computer 810, such as during start-up, can be stored in nonvolatile memory, while volatile memory can act as external cache memory to facilitate processing by the processor(s) 820, among other things.
Mass storage 850 includes removable/non-removable, volatile/non-volatile computer storage media for storage of large amounts of data relative to the memory 830. For example, mass storage 850 includes, but is not limited to, one or more devices such as a magnetic or optical disk drive, floppy disk drive, flash memory, solid-state drive, or memory stick.
Memory 830 and mass storage 850 can include, or have stored therein, operating system 860, one or more applications 862, one or more program modules 864, and data 866. The operating system 860 acts to control and allocate resources of the computer 810. Applications 862 include one or both of system and application software and can exploit management of resources by the operating system 860 through program modules 864 and data 866 stored in memory 830 and/or mass storage 850 to perform one or more actions. Accordingly, applications 862 can turn a general-purpose computer 810 into a specialized machine in accordance with the logic provided thereby.
All or portions of the claimed subject matter can be implemented using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to realize the disclosed functionality. By way of example and not limitation, the test generation system 100, or portions thereof, can be, or form part, of an application 862, and include one or more modules 864 and data 866 stored in memory and/or mass storage 850 whose functionality can be realized when executed by one or more processor(s) 820.
In accordance with one particular embodiment, the processor(s) 820 can correspond to a system on a chip (SOC) or like architecture including, or in other words integrating, both hardware and software on a single integrated circuit substrate. Here, the processor(s) 820 can include one or more processors as well as memory at least similar to processor(s) 820 and memory 830, among other things. Conventional processors include a minimal amount of hardware and software and rely extensively on external hardware and software. By contrast, an SOC implementation of processor is more powerful, as it embeds hardware and software therein that enable particular functionality with minimal or no reliance on external hardware and software. For example, the test generation system 100 and/or associated functionality can be embedded within hardware in a SOC architecture.
The computer 810 also includes one or more interface components 870 that are communicatively coupled to the system bus 840 and facilitate interaction with the computer 810. By way of example, the interface component 870 can be a port (e.g., serial, parallel, PCMCIA, USB, FireWire . . . ) or an interface card (e.g., sound, video . . . ) or the like. In one example implementation, the interface component 870 can be embodied as a user input/output interface to enable a user to enter commands and information into the computer 810, for instance by way of one or more gestures or voice input, through one or more input devices (e.g., pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer . . . ). In another example implementation, the interface component 870 can be embodied as an output peripheral interface to supply output to displays (e.g., CRT, LCD, LED, plasma . . . ), speakers, printers, and/or other computers, among other things. Still further yet, the interface component 870 can be embodied as a network interface to enable communication with other computing devices (not shown), such as over a wired or wireless communications link.
What has been described above includes examples of aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the disclosed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.
