Patent Application
20090327995
Modern computing environments are typically multi-threaded, employ advanced features such as asynchronous input/output, and often exist in a distributed environment. Traditional cyclic debugging processes struggle with such a complex environment and, as a result, the environment has become increasingly challenging for developers to debug.
One existing solution for de-bugging such a computing environment is a technique referred to as deterministic replay. Deterministic replay is a powerful approach for de-bugging multi-threaded and distributed applications. Deterministic replay can bring together all relevant states spread across numerous machines in a distributed system, removing non-determinism, and thus re-enabling the cyclic de-bugging process.
A second solution for de-bugging complex computing environments is a library based deterministic replay. The library based deterministic replay process utilizes a lot of what is referred to as replay-enabling code. However, these existing approaches require that each replay-enabling code be coded manually, which is not only tedious, but manually coding may also lead to the introduction of a multitude of errors during the de-bugging process. Therefore, there is a need for automated generation of replay-enabling code.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Methods and systems for automatically generating replay-enabling code are described. The automatic generating provides for faithful replay in a library-based deterministic replay system.
In one embodiment, an application program interface (API) function is intercepted by a virtual execution layer. The API function is then encapsulated by a code wrapper generated by annotation information. In addition, the annotation information also generates code snippets, which may be referred to as slots. The code wrapped API function is then converted into a flexible extension structure, where the flexible extension structure is plugged into the generated slots.
Automatic generation of replay enabling code reduces the need for hand-coding of API functions. Furthermore, the use of annotation-aware code generation increases the efficiency and robustness of a library-based deterministic replay system.
The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items.
This disclosure is directed to techniques for automatically generating a replay-enabling code. The technique requires a code template programmed by an operating system developer. Then, utilizing an application programming interface (API) annotation, either standard or user-defined, a customized code is automatically generated for every specific API.
The following discussion of an exemplary system provides the reader with assistance in understanding ways in which various subject matter aspects of the system, methods, and computer program products may be employed. The system described below constitutes but one example and is not intended to limit application of the subject matter to any one particular operating system.
As depicted in the replay tool 102, there are upper application(s) 104 which communicates with the underlying library(ies) 106 and the operating system(s) 108 via a multitude of application program interface (API) functions 110(1)-110(n). The API functions exist in what may be referred to as a virtual execution layer 112. The virtual execution layer 112 represents a natural boundary between the upper application 104 and the underlying supporting infrastructure including, without limitation, the library 106 and the operating system 108.
Separation between the application space 200 and the system space 202 is typically done at an entry and a return point of intercepted API functions 110(1)-110(n) by setting a thread specific flag. An exception to this typical separation format is user functions that are invoked from the system space 202. These functions are referred to as exceptional control flow. These functions are typically callback events, discussed below, and the functions are registered either explicitly or implicitly. Examples of these include but are not limited to, asynchronous ReadFileEx with callback events, and message handler registration in many distributed systems. Message handler registrations may include, for example, per-thread dynamic-link library (DLL) initializations on Windows. While a routine is designated when the thread is created, the actual execution of the thread will first invoke entry points of an in-process DLL. This kind of callback registration is done automatically at run time by the underlying operating system.
To utilize the replay tool 102, API functions 110(1)-110(n) should be intercepted, wrapped and converted to a SignalEx structure. Today's operating system often has a rich set of system API functions 110(1)-110(n), which may include more than 1000 functions. Therefore, this can be a challenging task.
In one implementation, forming of the virtual execution layer 112 allows the replay tool 102 to intercept the API functions 110(1)-110(n) using a technique referred to as a detour. However, in alternative implementations other techniques may be used to intercept the API functions. A detour is a library for intercepting functions. The detour operates by replacing the first few instructions of the target function with a jump to the user-provided detour function. Detours are typically inserted at the time of execution. The code of the target function is modified in memory, not on a disk, therefore permitting interception of the API functions 110(1)-110(n) at a very fine level. For example, the procedures in a dynamic link library (DLL) can be detoured in one execution of an application, while the original procedures are not detoured in another execution running at the same time. In general, techniques used in the detour library work regardless of the method used by the upper level application 102 or system code to locate the target function.
Hand coding of these wrappers and slots is tedious and error-prone. Therefore, instead of implementing the wrappers and slots on each API function, programmers annotate function prototypes and the wrappers as well as slots are then automatically generated according to these annotations. Annotation-aware code generation enables large scale interception of API functions 1108(1)-110(n) and code generation for log and replay, further enabling the flexibility to redefine the interception layer.
Most Windows® functions are well annotated in a Standard Annotation Language (SAL) as shown in recent Windows® Platform Software Development Kits. The functions concisely describe various aspects of attributes as well as parameters of the API functions 110(1)-110(n) and form the basis for annotation-aware code generation. For example, SAL, may include, without limitation:
In this example, “in” indicates that parameters s, len and flags are input parameters; “out” indicates that buf is the output buffer that must be logged for replay, with “bcount” to specify its initial length is len and the result length is the return value.
Special attention may be required for exchanging data across spaces, for example, application space 200 and system space 202. For most API Functions 110(1)-110(n) a caller is responsible to prepare and to manage the buffer, and the API Function 110(1)-110(n) just needs to fill the buffer. Therefore, a user would need to copy the content of the buffer to the log. However, other functions return a buffer that is allocated internally. For example, inet_ntoa returns a string for a given network address, in which the buffer is maintained by the callee rather than the caller.
This presents a few challenges. First, it is generally unsafe to re-execute these API Functions 110(1)-110(n) at replay time and hope that the outputs match exactly with the original run. Second, even if the content of the execution is reproducible, the call can return a pointer that may differ from the original run. If an application were to use this pointer as an input, state corruption would inevitably occur. This problem can be addressed by allocating a shadow copy, permitting manual or automatic backup copies of that specific point in time. Returning the shadow copy to the application at both the logging and replay time circumvents potential non-determinism issues caused by these API functions. The annotation xpointer instructs the script to generate appropriate codes. Specifically:
XPointer(global)/xpointer (tls) are used to indicate that the buffer is taken from a global/per-thread internal buffer space.
Table 2 provides a summary of annotations that are used in addition to those provided by SAL.
Replay tool 102 operates as an event-based tool. For example, as discussed above, the wrapped API functions 302(1)-302(n) direct the execution of thread 304 into the correct slot within the signal-slot process 306. Typically, the execution of the thread 304 is viewed as a succession of three types of events. These three events include, includes, but is not limited to, an API event, a continuation event, and a callback event. The API event is an invocation of an intercepted API functions 110(1)-110(n). The API event segments the thread execution into the continuation events. Some of these API functions may take callback routines that will be executed at some future points, and the invocations are the callback events.
A multi-threaded, distributed application is a collection of these three events from the various events running on the distributed computing devices. The task of logging these events includes at least two steps. First, numbering of the events, and second, recording the output of the API events such that the replay tool 102 can process these events in increasing order while feeding the outputs of the API events from the log. This ensures that the internal state of the application can be faithfully recreated as dictated by the application logic.
The events are numbered by assigning each event a 64-bit integer that is a logical clock. Logical clocks are assigned within a process, without limitation, by one of two main approaches. First, logical clocks are assigned through use of a customized scheduler which defines scheduling points at a boundary of the intercepted API functions 110(1)-110(n). The second approach begins with each thread inheriting a logical clock from its creator. The logical clock is then modified to reflect a relationship among events by capturing the relationship between the various API events that access the same resource. A shadow memory block is allocated behind each resource such that it may store, without limitation, the thread ID and the logical clock of the last API event that accessed the resource. When an API event accesses a resource, the corresponding logical clock is updated with the maximum of either its own clock or that of the last logical clock value recorded on the shadow memory block, therefore processing events in the order as determined by the logical clock.
Logical clock values may also be assigned across processes using a layered service provider. A layered service provider implements only higher-level communication functions while relying on an underlying transport stack for the actual exchange of data with a remote endpoint. Such communication may, for example and without limitation, take place by transferring messages through the use of a socket. A socket is an identifier for a particular service on a particular node of a network. The socket includes a node address and a part number, identifying the service. The layered service provider will build a filter and a message processing layer. All socket based messages with travel through this layer, whereby a logical clock is embedded in the outgoing message and extracted as it enters. Such a process is transparent to the application.
As illustrated in
Memory 504 may store programs of instructions that are loadable and executable on the processor 502, as well as data generated during the execution of these programs. Depending on the configuration and type of computing device, memory 504 may be volatile (such as RAM) and/or non-volatile (such as ROM, flash memory, etc.). The system may also include additional removable storage 506 and/or non-removable storage 508 including, but not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable medium may provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for the communication devices. Memory 504, removable storage 506, and non-removable storage 508 are all examples of the computer storage medium. Additional types of computer storage medium that may be present include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computing device 103.
Turning to the contents of the memory 504 in more detail, may include an upper level application 104, an operating system 514, one or more replay tools 102. For example, the system 500 illustrates architecture of these components residing on one system or one server. Alternatively, these components may reside in multiple other locations, servers, or systems. For instance, all of the components may exist on a client side. Furthermore, two or more of the illustrated components may combine to form a single component at a single location. In one implementation, the memory 504 includes the replay tool 102, a data management module 514, and an automatic module 516. The data management module 514 stores and manages storage of information, such as images, ROI, equations, and the like, and may communicate with one or more local and/or remote databases or services. The automatic module 516 allows the process to operate without human intervention. The system 500 may also contain communications connection(s) 518 that allow processor 502 to communicate with servers, the user terminals, and/or other devices on a network. Communications connection(s) 518 is an example of communication medium. Communication medium typically embodies computer readable instructions, data structures, and program modules. By way of example, and not limitation, communication medium includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable medium as used herein includes both storage medium and communication medium. The system 500 may also include input device(s) 520 such as a keyboard, mouse, pen, voice input device, touch input device, etc., and output device(s) 522, such as a display, speakers, printer, etc. The system 500 may include a database hosted on the processor 502. All these devices are well known in the art and need not be discussed at length here.
Although embodiments for automatic generation of code have been described in language specific to structural features and/or methods, it is to be understood that the subject of the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementations.
This application is related to commonly-filed application Ser. No. ______, entitled “Space Separation for a Library Based Record and Replay Tool”, both of which are commonly assigned to Microsoft Corp., the disclosure of which is incorporated by reference herein.
