Patent Application
20150143180
Software often suffers from memory and performance issues. For example, a software application may have a memory leak in which the software application requests memory but does not free the memory. Over time, this can lead to the software application consuming all or a significant portion of the memory available on a system. Similarly, the performance of a software application may degrade or vary significantly over time. These issues are frequently difficult to detect and fix.
The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.
Briefly, aspects of the subject matter described herein relate to software validation. In aspects, code may be instrumented to generate certain records upon execution. The code may be further instrumented to generate start and stop records that correspond to the start and stop events of a scenario of a program. The start and stop event records allow correlation of the scenario with other records written to the log. With the correlation and appropriate instrumentation, a tool may determine performance, memory usage, functional correctness, and other characteristics of a program at the granularity of the scenario.
This Summary is provided to briefly identify some aspects of the subject matter that is further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
The phrase “subject matter described herein” refers to subject matter described in the Detailed Description unless the context clearly indicates otherwise. The term “aspects” should be read as “at least one aspect.” Identifying aspects of the subject matter described in the Detailed Description is not intended to identify key or essential features of the claimed subject matter.
The aspects described above and other aspects of the subject matter described herein are illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “or” is to be read as “and/or” unless the context clearly dictates otherwise. The term “based on” is to be read as “based at least in part on.” The terms “one embodiment” and “an embodiment” are to be read as “at least one embodiment.” The term “another embodiment” is to be read as “at least one other embodiment.”
As used herein, terms such as “a,” “an,” and “the” are inclusive of one or more of the indicated item or action. In particular, in the claims a reference to an item generally means at least one such item is present and a reference to an action means at least one instance of the action is performed.
Sometimes herein the terms “first”, “second”, “third” and so forth may be used. Without additional context, the use of these terms in the claims is not intended to imply an ordering but is rather used for identification purposes. For example, the phrases “first version” and “second version” do not necessarily mean that the first version is the very first version or was created before the second version or even that the first version is requested or operated on before the second version. Rather, these phrases are used to identify different versions.
Headings are for convenience only; information on a given topic may be found outside the section whose heading indicates that topic.
Other definitions, explicit and implicit, may be included below.
Aspects of the subject matter described herein are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations that may be suitable for use with aspects of the subject matter described herein comprise personal computers, server computers—whether on bare metal or as virtual machines—, hand-held or laptop devices, multiprocessor systems, microcontroller-based systems, set-top boxes, programmable and non-programmable consumer electronics, network PCs, minicomputers, mainframe computers, personal digital assistants (PDAs), gaming devices, printers, appliances including set-top, media center, or other appliances, automobile-embedded or attached computing devices, other mobile devices, phone devices including cell phones, wireless phones, and wired phones, distributed computing environments that include any of the above systems or devices, and the like. While various embodiments may be limited to one or more of the above devices, the term computer is intended to cover the devices above unless otherwise indicated.
Aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. Aspects of the subject matter described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
Alternatively, or in addition, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
With reference to
The processing unit 120 may be connected to a hardware security device 122. The security device 122 may store and be able to generate cryptographic keys that may be used to secure various aspects of the computer 110. In one embodiment, the security device 122 may comprise a Trusted Platform Module (TPM) chip, TPM Security Device, or the like.
The computer 110 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 110 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media.
Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EEPROM, solid state storage, flash memory or other memory technology, CD-ROM, digital versatile discs (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer 110. Computer storage media does not include communication media.
Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media, discussed above and illustrated in
A user may enter commands and information into the computer 110 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball, or touch pad. Other input devices (not shown) may include a microphone (e.g., for inputting voice or other audio), joystick, game pad, satellite dish, scanner, a touch-sensitive screen, a writing tablet, a camera (e.g., for inputting gestures or other visual input), or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB).
Through the use of one or more of the above-identified input devices a Natural User Interface (NUI) may be established. A NUI, may rely on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and the like. Some exemplary NUI technology that may be employed to interact with a user include touch sensitive displays, voice and speech recognition, intention and goal understanding, motion gesture detection using depth cameras (such as stereoscopic camera systems, infrared camera systems, RGB camera systems, and combinations thereof), motion gesture detection using accelerometers/gyroscopes, facial recognition, 3D displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, as well as technologies for sensing brain activity using electric field sensing electrodes (EEG and related methods).
A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 195.
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 may include a modem 172, network card, or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
As mentioned previously, software may suffer from various issues.
In some implementations, the components described in conjunction with
As used herein, the term component may be read in alternate implementations to include hardware such as all or a portion of a device, a collection of one or more software modules or portions thereof, some combination of one or more software modules or portions thereof and one or more devices or portions thereof, or the like. In one implementation, a component may be implemented by structuring (e.g., programming) a processor (e.g., the processing unit 120 of
For example, the components illustrated in
An exemplary device that may be configured to implement one or more of the components of
In one implementation, a component may include or be represented by code. Code includes instructions that indicate actions a computer is to take. Code may also include data, resources, variables, definitions, relationships, associations, and the like that include information other than actions the computer is to take. For example, code may include images, Web pages, HTML, XML, other content, and the like.
Actions indicated in code may be encoded in a source code language, intermediate language, assembly language, binary language, another language, some combination of the above, and the like.
Code may be executed by a computer. When code is executed by a computer, this may be called a process. The term “process” and its variants as used herein may include one or more traditional processes, threads, components, libraries, objects that perform tasks, and the like. A process may be implemented in hardware, software, or a combination of hardware and software. In an embodiment, a process is any mechanism, however called, capable of or used in performing an action. A process may be distributed over multiple devices or a single device. Code may execute in user mode, kernel mode, some other mode, a combination of the above, or the like. A service is another name for a process that may be executed on one or more computers.
When the term “thread” is used herein, this term is to be read as a traditional software thread. A thread typically executes in the context of a process and multiple threads that execute in the context of a process may share state, memory, and other resources.
Although the terms “client” and “server” are sometimes used herein, it is to be understood, that a client may be implemented on a machine that has hardware and/or software that is typically associated with a server and that likewise, a server may be implemented on a machine that has hardware and/or software that is typically associated with a desktop, personal, or mobile computer. Furthermore, a client may at times act as a server and vice versa. At times, two or more entities that more frequently act as a client or server may concurrently be peers, servers, or clients. In an embodiment, a client and server may be implemented on the same physical machine.
Furthermore, as used herein, each of the terms “server” and “client” may refer to one or more physical or virtual entities, one or more processes executing on one or more physical or virtual entities, and the like. Thus, a server may include an actual physical node upon which one or more processes execute, a virtual node upon which one or more processes execute, a service executing on one or more nodes, a group of nodes that together provide a service, and the like.
For simplicity in explanation, some of the actions described below are described in a certain sequence. While the sequence may be followed for some implementations, there is no intention to limit other implementations to the particular sequence. Indeed, in some implementations, the actions described herein may be ordered in different ways and may proceed in parallel with each other.
Turning to
The sources 205-207 provide log data for executed code. The sources 205-207 may correspond to different layers of a platform upon which an application executes. For example, a source may include instrumented code of a software application, an execution environment (sometimes referred to as a runtime), a rendering engine, system code (e.g., operating system code, file system code, and other system code), server code (e.g., code that responds to requests generated by the software application), portions of the application that were written in different languages, a host upon which the application executes, or the like.
The sources 205-207 may provide log data across multiple versions of one or more of the sources indicated above. For example, the source 205 may provide log data for an application executed in a first runtime version, the source 206 may provide log data for an application executed in a second runtime version, and so forth.
The logging managers 210-212 write logs to the memory 220. The logging managers 210-212 may include application programming interfaces (APIs) that the sources 205-207 call to provide log data. A logging manager may be included as part of a source (e.g., instrumented code of a source may write log records to the memory 220). A logging manager may receive log requests with a variety of data and may output log records of a fixed format (e.g., supplying and formatting fields as needed in a defined manner).
Logs may be stored in the memory 220. The memory 220 may include any storage media capable of storing data. The memory 220 may comprise volatile memory (e.g., RAM), nonvolatile memory (e.g., a hard disk), some combination of the above, and the like and may be distributed across multiple devices. The memory 220 may be external, internal, or include one or more components that are internal and one or more components that are external to computer(s) hosting the analysis system 202.
In one implementation, a log may include one or more records. Each record may include one or more data elements. A record may include one or more different data elements than another record. Some exemplary data elements that may be included in a log record include:
1. Timestamp: A timestamp may include a real time as obtained or maintained by a computer, a counter of a computer that corresponds to real time, a counter of a computer that increases over time but that does not increase proportionate to real time (e.g., each count may correspond to a different length of real time), a day, a month, a year, some combination of the above, or the like. If the system is capable, a high-precision timestamp may be used.
2. A process identifier. A process identifier may identify a process for which a log record was written.
3. A thread identifier. A thread identifier may identify a thread for which a log record was written.
4. A memory allocated value. A memory allocated value may indicate how much memory was allocated (delta or absolute) when the log record was created.
5. Objects allocated. The actual objects that are allocated may be output to a log. The memory allocated for each object may also be output to a log.
6. A scenario identifier. A scenario identifier identifies a testing scenario. A scenario may include one or more functions, events (e.g., clicking of a button or other user interface or system event), program statements, program steps, program actions, or the like. In one implementation, a scenario may be defined as all code executed between two selected statements in the code. The scenario may further be defined by process identifier and thread identifier as indicated herein. In one implementation, scenario records for a given scenario may include all log records that occur after a start scenario record and before a corresponding stop scenario record and that further include the process identifier and thread identifier identified by the start scenario record
The term “function” as used herein may be thought of as a portion of code that performs one or more tasks. Although a function may include a block of code that returns data, it is not limited to blocks of code that return data. A function may also perform a specific task without returning any data. Furthermore, a function may or may not have input parameters. A function may include a subroutine, a subprogram, a procedure, method, routine, or the like.
A testing scenario may be chosen to correspond to an end-user experience. For example, a testing scenario may be chosen to correspond to a user clicking a back button in a Web browser, a user clicking a save button of an application, a user gesturing on a touch-sensitive surface, a user scrolling through a document, or any other user interaction.
7. A start identifier. A start identifier may be used to indicate a start of a scenario.
8. A stop identifier. A stop identifier may be used to indicate an end of a scenario.
9. A call stack that exists when a logging statement occurs.
10. Values and names of one or more local variables that exist when a logging statement occurs.
11. Values and names of one or more global variables available when a logging statement occurs.
12. Hints as described below.
13. Whether a test scenario passed.
A configuration option may allow selection of what data is placed in a log record. For example, an option may indicate minimalistic logging, maximal logging, normal logging, a list of fields to log, or the like.
The examples above are not intended to be all-inclusive or exhaustive. Indeed, based on the teachings herein, those skilled in the art may recognize other data that may be logged without departing from the spirit or scope of aspects of the subject matter described herein.
Log records from multiple sources may be intermingled in the memory 220. In particular, log records from multiple processes and/or multiple threads may be written in the memory 220. Furthermore, although one memory is shown in
In one implementation, the analyzer 225 may perform the following exemplary steps to identify log records associated with a given scenario:
1. Scan the memory 220 for a start event record that includes an identifier associated with the scenario.
2. Select the next log record of the memory 220. In multi-processor/thread environments, if the log record includes the same process identifier and thread identifier as the start event record for the scenario, the log record is part of the scenario.
3. Repeat 2 until a stop event record is found that includes the identifier associated with the scenario. A stop event for the scenario may correspond to a selected event of a program (e.g., a function is called, returned from, a button is clicked or released, another event occurs, or the like) or to another scenario (e.g., the start or end event of another scenario occurs).
The actions above have the effect of correlating a scenario with log records that occur during the scenario. For example, referring to
The start and stop events shown in section 310 define intervals of interest. Logged events that occur in the intervals may be used to determine performance, memory usage, and the like for repetitions of a scenario. Start and stop events may be logged each time a scenario occurs. The scenario start and stop events may correspond, for example, to clicking a button in a user interface.
In a single log, the log events shown in section 305 and section 310 may be combined. If the log events in sections 305 and 310 are in separate logs, the timestamp of each log event may be used to ensure that the events of section 305 fall between the events of section 310.
While the application event log records may be useful to show how much memory is being allocated with each instrumented allocation statement, without the start and stop events of section 310, the application event records may be hard or impossible to identify as corresponding to an identified user action (e.g., clicking a button) represented by the start and stop events of section 310. Logging the start and stop events of this scenario allows correlation between application events of interest and the scenario.
The logs for a scenario may be repeated multiple times in the memory 220. Differences across iterations may be used by the analyzer 225 to calculate statistics regarding the scenario. For example, data extracted from the log records written in each iteration of the scenario may be used to determine whether memory usage, performance, or some other execution characteristics are changing across iterations. To collect and analyze differences across iterations, steps 1-3 may be repeated until all log records have been examined.
The analyzer 225 may perform similar analysis on other characteristics written in a log. For example, the analyzer 225 may perform analysis on the duration the scenario across iterations.
As another example, the analyzer 225 may perform analysis on throughput of servicing client requests. For example, during execution, an application may make requests of a server component. A log may be written logging these requests and responses thereto. Afterwards, analysis may be performed to obtain statistics. Some exemplary throughput statistics include: maximum requests per time period, mean requests per time period, standard deviation, and trend slope.
Returning to
These statistical values may be outputted in a form suitable for viewing. For example, statistical values may be outputted in a Web page or other document that shows the statistical values.
Below is a table that shows some exemplary values for the bytes used by an application and the bytes used by a runtime across several iterations of a scenario:
A value may be outside of an expected range. For example, it may be expected that there will be no increase in memory consumption over multiple iterations of a scenario. In one implementation, a scenario may indicate or be associated with a hint that indicates expected behavior. If actual behavior falls outside of the expected behavior, the unexpected behavior may be highlighted (e.g., via bolding, coloring, background coloring, flashing, some other highlighting, or the like).
Some exemplary hints include:
1. That a scenario is expected to have a constant duration and memory growth when repeated.
2. That a scenario is expected to have a constant duration and no memory growth when repeated.
3. That a scenario is expected to have variable duration and no memory growth when repeated.
4. That a scenario is expected to have variable duration and variable memory growth.
5. That a scenario will use less than a specified amount of memory.
6. That a scenario will open the same number of files when repeated.
The examples above are not intended to be all-inclusive or exhaustive of what hints may be provided. Based on the teachings herein, those skilled in the art may recognize other hints that may be provided without departing from the spirit or scope of aspects of the subject matter described herein.
Below is a table that shows some exemplary values for duration for a scenario executed multiple times:
The table above includes a name of each scenario, the results (i.e., pass) of the scenarios, number of iterations, max duration in milliseconds, mean duration in milliseconds, and standard deviation. Other values (e.g., slope of trend line, memory usage statistics, or the like) may also be shown in the table without departing from the spirit or scope of aspects of the subject matter described herein.
There are other ways to obtain the records associated with a scenario. For example, with a query language, records not including the process identifier and thread identifier may be filtered out before processing a log. For example, a query language may be used to obtain a result set that has irrelevant records filtered out. After the irrelevant records are filtered out, each record in a result set may be related to the scenario. In this example, the following exemplary steps may be performed to identify log records associated with a given scenario:
1. Search the result set for a start event record.
2. Select the next record of the result set. This record is part of the scenario.
3. Repeat step 2 until a corresponding stop event record is found. In one implementation, a corresponding stop event record includes the same process identifier and thread identifier as the start event record.
4. Repeat steps 1-3 until all records of the result set have been examined.
There is no intention to limit the ways of correlating log records with a scenario to the examples above. Indeed, based on the teachings herein, those skilled in the art may recognize many other ways of obtaining the records associated with a scenario without departing from the spirit or scope of aspects of the subject matter described herein.
The output manager 230 may prepare reports that include data generated by the analyzer 225. For example, the output manager 230 may output graph data that indicates iteration, delta memory usage per iteration, and a trend line of memory usage over iterations of the test scenario (as illustrated in
The client 235 may include any entity that interacts with the analysis system 202 to obtain output data. For example, the client 235 may be implemented as a Web browser that sends requests for analysis data to the analysis system 202 and receives Web pages in response. As another example, the client 235 may be implemented as proprietary software on a computer that requests analysis data from the analysis system 202 and outputs formatted data derived from the analysis system 202 on an output device such as a display.
Turning to
At block 515, a start event record is located of a test scenario. The start event record may indicate, for example, an identifier and a start timestamp. For example, referring to
At block 520, scenario records within the log are identified. For example, referring to
At block 525, a stop event record for a scenario is located. The stop event record may indicate, for example, an identifier of the scenario and a stop timestamp. For example, referring to
At block 530, a search is performed for another start event record of the scenario. For example, referring to
At block 535, if another start record is found, the actions continue at block 520; otherwise, the actions continue at block 545.
At block 540, statistics are determined for the scenario. Statistics may be determined and updated at any time during the processing of a log. For example, referring to
At block 545, other actions, if any, are performed. Other actions may include, for example, preparing a report from the determined statistics. In preparation for preparing the report, a hint may be obtained that indicates expected behavior as the scenario repeats. In one example, the report may contain a graph such as the graph shown in
Turning to
At block 615, the report is received. For example, referring to
At block 620, the report may be outputted. For example, referring to
At block 625, other actions, if any, may be performed.
Although some of the discussion above has focused on performance and memory usage, the same concepts may also be applied to other characteristics. Some exemplary characteristics include how many files are opened, how many Web requests are made, how many functions are called, other characteristics, and the like.
As can be seen from the foregoing detailed description, aspects have been described related to software validation. While aspects of the subject matter described herein are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit aspects of the claimed subject matter to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of various aspects of the subject matter described herein.
