A portion of the disclosure of this patent document contains material, which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by any one of the patent disclosure, as it appears in the United States Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.
This invention relates generally to the testing of software applications, and more particularly to the ensuring that a graphical user interface element (UI element) is ready for interaction.
Almost, if not all, modem operating systems are multi-threaded. Furthermore, more and more systems allow concurrent applications, each with their own threads, to be running using multi-processors. At the same time, the rise of graphical user interface applications which use the threads, have allowed users to interface with both the operating system and whatever applications may be running on it in an astounding number of ways. For example, multiple applications, each application with multiple windows, can be running simultaneously. The user is presented with an almost unlimited number of paths through the feature sets. Using the mouse or keyboard input the user can impulsively switch from window to window, and treenode to test box.
When testing applications with graphical user interfaces (GUIs), a tester must take both the user-driven nature of GUIs and the many choices offered to the user at any time-the multiple paths problem-into account. However, sometimes such needs are contradictory. For example, one solution to the multiple paths program is to automate the GUI testing. As automated testing programs can be run at computer speed, many more pathways through a GUI can be tested than is reasonable when using human testers. But, computers and humans each have their own strengths, and one thing humans excel at is the ability to discern the difference between a minor hiccup in a program and an actual code bug.
Due to the complex interaction between the many threads running on even a modest GUI application and the interaction between those threads, the operating system threads, and the threads of any other applications running, certain actions may fail not because of any underlying problems with the software, but merely because of timing issues. A human tester will most likely ignore a mouse click that does not select an object, but an automated tester will record such an event as a failure; a thorny test error that is impossible to recreate and difficult to understand.
Referring to prior art
Referring to prior art
Bring up control window 100 ( )
Select(Item 102C(1));
This code will often fail because the control window takes awhile to draw all of the arrayed items, such that the Select code line is run before Item 102C(1) appears on the window, and thus, it cannot yet be selected. With reference again to
The length of time between T1 and T3 is very difficult to predict. Developers, when presented with the problem of timing failures, have a tendency to solve it by forcing a program to sleep for a long enough period of time that the operating system will have time to perform the action. One or two such sleep statements may not cause a problem, but they very quickly add up and can cause serious degradation in program execution time. Developers also tend to overestimate the amount of time necessary to sleep, for fear of not waiting long enough, so sleeps of as long as 10 seconds are occasionally seen. Furthermore, a sleep of appropriate length on one machine may be much too short or much too long on another that runs at a different speed.
It can be seen that when a function with an embedded sleep statement is called by another function that also contains a sleep, the time required to execute the program can quickly cascade out of control. For example, the time taken to run regression tests has been shown to triple with the incorporation of sleeps to solve the synchronization problem described above.
One prior art solution comprises locating the target test interaction within a message queue, and waiting until the target test interaction has been processed, at which point the user is notified. However, due to the interaction with uncontrollable background threads, this is reliable only about half of the time.
Another prior art solution involves setting the “WH_FOREGROUNDIDLE” hook, a procedure that is called when an application's foreground thread is about to become idle. However, using this requires that the test code be loaded into the process space of the target application, often undesirable, as it may change the parameters of the target process and can lead to the introduction of new bugs in the target application caused by the testing process itself. It should go without saying that a testing application should not be the source of errors that appear to be caused by the program under test.
As described herein, various new and non-obvious apparatus and methods are disclosed to ensure that a UI element is ready to perform a requested action and this action is performed successfully. If the requested action is not performed, an error condition is generated in such a way that a test program knows the nature of the error. The disclosed exemplary apparatus and methods should not be construed as limiting in any way. Instead, the present disclosure is directed toward novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and subcombinations with one another. The methods are not limited to any specific aspect, feature, or combinations thereof, nor do the disclosed methods require that any one or more specific advantages be present or problems be solved.
In one implementation, a UI element is found, and a requested action that is to be performed on the UI element is chosen. Quite often, the requested action is that the UI element is selected, but it may be any action that can be performed by a user emulation test program. A time-out criterion is determined that will generate an error message informing the calling program that there has been a failure due to a synchronization problem. A common criterion is that too much time has been spent waiting for the action to be performed. Then, the requested action is attempted to be performed. If the action is unable to be performed, then the time-out criterion is checked; if it has not been satisfied, the program sleeps, and then attempts to perform the action again.
In one implementation, when attempting to perform the requested action, the UI element's associated window and thread are determined. Then, prior to attempting to perform the requested action, the program pauses until the associated window is ready to accept user input. For instance, this may comprise pausing until the thread associated with the window is the foreground thread, by pausing until the window associated with the object is enabled, or by pausing until the window associated with the object has focus.
To improve upon this implementation, performance counters associated with the target process can be checked at two different times. If the two time-spaced readings are essentially the same, then it is assumed that the target process is ready for a user input and the action is attempted to be completed. If the values of the two readings are too far apart, then, the process pauses, and—the cycle repeating—again takes the two readings.
To improve the synchronization ability still further, the action is requested to be performed, then a check is made to determine if the performance occurred. If not, the process pauses, and another attempt is made to perform the action. This process continues until either the action is successfully performed, or the time-out criterion is met. If the time-out condition is met, an error condition is generated, which allows the test program to determine that the failure was the result of a synchronization problem.
In an exemplary embodiment, several different pause types are provided. For example, the process may pause for a specific time, until an event occurs, or may return immediately.
Additional features and advantages will be made apparent from the following detailed description of various embodiments that proceeds with reference to the accompanying drawings.
Described embodiments relate to techniques and tools for synchronizing a test application with a target application which the test application does not control. The various techniques and tools can be used in combination or independently.
With reference to
A computing environment may have additional features. For example, the computing environment (200A) includes storage (240A), one or more input devices (250A), one or more output devices (260A), and one or more communication connections (270A). An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing environment (200A). Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment (200A), and coordinates activities of the components of the computing environment (200A).
The storage (240A) may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information and which can be accessed within the computing environment (200A). The storage (240A) stores instructions for the software (280A) implementing the synchronizer.
The input device(s) (250A) may be a touch input device such as a keyboard, mouse, pen, trackball, a voice input device, a scanning device, or another device that provides input to the computing environment (200A). For audio or video encoding, the input device(s) (250A) may be a sound card, video card, TV tuner card, or similar device that accepts audio or video input in analog or digital form, or a CD-ROM or CD-RW that reads audio or video samples into the computing environment (200A). The output device(s) (260A) may be a display, printer, speaker, CD-writer, or another device that provides output from the computing environment (200A).
The communication connection(s) (270A) enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired or wireless techniques implemented with an electrical, optical, RF, infrared, acoustic, or other carrier.
The techniques and tools can be described in the general context of computer-readable media. Computer-readable media are any available media that can be accessed within a computing environment. By way of example, and not limitation, with the computing environment (200A), computer-readable media include memory (220A), storage (240A), communication media, and combinations of any of the above.
The techniques and tools can be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing environment (200A) on a target real or virtual processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing environment.
Automated testing of GUIs often produces spurious failures due to synchronization problems with the myriad threads running at any given time on an operating system. One embodiment disclosed herein is an improved method to synchronize UI elements specifically to ensure that a target UI element will not fail when attempting to accept user input.
As used herein, the terms control, element, item and object are interchangeable, and also encompass ideas such as container and component. The idea behind all such listed words embraces any viewable object, such as listboxes, combo boxes, tree structures, radio buttons, calendars, windows, forms, panels, and combinations thereof. New implementations of viewable objects are being constantly created and the embodiments disclosed embrace viewable elements that have not been formally given a name.
The meaning of the term synchronization as used here is slightly different from the common meaning of this term. Usually, the term synchronization assumes a 100% guaranteed result, that is, that the given objects are synchronized or that an error state occurs. For instance, when Windows API call WaitForSingleObject( ) is called, it will either return when the object enters the desired state or when a time-out interval has lapsed. That is, it always works or throws a known error condition unless the hardware malfunctions. In the case of UI-based applications, however, many operating systems provide no mechanisms that can be used to synchronize two or more UI elements. This allows synchronization errors to occur that are otherwise unmarked. We use the term synchronization here to describe a mechanism or a combination of different mechanisms where a small failure rate is acceptable when the reason for the failure is almost always known and communicated. The term “select item” refers to a user interaction or to a test program that mimics user interaction with a UI element using a mouse, a keyboard, or another user input device.
To ensure that a user emulation test program either successfully synchronizes with a target program or an error message is generated, which indicates that an apparent failure was due to a synchronization problem, the following systems and processes are provided.
With reference to
In some embodiments, the user emulation test program (202B) should not verify states of all background threads (212B, 214B) since some of them may always be in the busy state and a synchronization mechanism may always raise a time-out error condition. The target UI element (218B) has, as a direct ancestor, a window (216B) associated with it. When this window is disabled, the target element will also be disabled. Similarly, the ancestor window must, at a minimum, have foreground focus for the target element to be able to be selected. The target element may itself be a window owned by another ancestor window.
The ancestor window (216B) of the target element (218B) has an associated thread (222B). This thread, in some embodiments, can be referred to as an ancestor of the window (216B) and of the UI element (218B).
The computer environment (200B) referred to above can be more fully understood with reference to
An exemplary embodiment is described in the flowchart of
Once an element, a time-out criterion, and an action for the element are selected, the action is carried out, as can be seen at process block 308A. But, it is not assumed that the action was successful. Rather, the process checks to determine if the action was carried out successfully. If it was, the process returns control to the calling entity. If the action was not successful, then the time-out criterion is checked. If the time-out criterion has been met, then an error condition is generated, as shown at process block 314A. The result might be an error message sent to a predefined location, a notation in a log, or another method of indicating that a synchronization error has occurred.
If the time-out criterion has not been met, then repeated attempts are made to perform the action either until the time-out criterion is met or until the action is successfully performed. This ensures that even though not every action may be successful, there will be no (or very few) mysterious error conditions caused by synchronization failures.
In certain embodiments, to ensure that a UI element has successfully performed a desired action, with respect to
If the user action is not performed, then the program waits until performance counters associated with the the target process have substantially the same readings at two different times, as shown at process block 306B. Once the performance counters have similar-enough readings it is assumed that the user input element window (or a relative) is ready for a user input, and the program again attempts to make the UI element perform the action.
If the attempt fails again, as shown at process block 310B, the action is again attempted to be performed, and is then checked to see if the performance was successful. If the attempt is not successful, it pauses, and tries again until the action is performed or until some time-out criteria is met.
At process block 402, the window associated with a target UI element is determined. This window can be referred to as an ancestor or a direct ancestor window of the target UI element. In some operating systems, when a displayable UI element is created, a window handle is also created. Among other functions, this window handle identifies the visual aspects of the object to the operating system and it also associates the object with its specific window. If the target UI element does not possess a window handle, then the closest ancestor with a window handle is located, and the window associated with that handle is considered the target element's window.
At process block 404, it is determined if the window which was located in process block 402 is enabled. If so, then the process continues at process block 408. If not, the process continues at process block 406. When a window is enabled, it is able to receive keyboard/mouse focus; it can be selected. Disabled windows ignore all attempted input. They cannot be selected by a user (or a user emulation test program) and are often, though not always, visually distinct from enabled windows; a common visual marker is that they are grayed. At process block 406 the process sleeps; this process is explored more thoroughly with reference to
At process block 408, it is determined if the window has foreground focus. If it does, then the process continues at process block 412. If not, the process is continued at process block 410, where the process first sleeps, and then again checks for foreground focus. Threads are, in many operating systems, the basic object that executes instructions on a processor. In such systems, each window has a thread associated with it. In multi-threaded operating systems, foreground focus ensures that the thread that created a specific window is the foreground thread; that is, the thread's windows are on top of the screen in the sense that they are not behind or overlapped by other application windows, and they can be selected by a user.
At process block 412, the window thread is checked to see if it is in a “wait for user input” state. If so, then the process continues at process block 418. If not, the process continues at process block 414. Each thread can be in many states, one of which, commonly, is “waiting”. Other common states are “running”, “ready”, “standby”, etc. The waiting state, typically, is composed of many sub-states, one of which is “waiting for user input” which is sometimes also called “waiting for a user request”. In Windows, the processor counter “Thread State” has the value 5, “waiting”; and the processor counter “Thread Wait Reason” has the value 6 or 13, “waiting for a user request”. In an exemplary embodiment, the process counter “Thread State” should have the value “5”, and the process counter “Thread Wait Reason” should have either value “6” or “13”.
At process block 414, it is decided if the thread state could not be determined. If so, this section of the process is considered to have failed, and in some embodiments the process continues as shown at
At process block 418, the requested action is performed on the target UI element. In an exemplary embodiment, the action entails selecting the UI element. If an error condition results—if an exception is thrown—then the action has failed. If no error condition occurs, then the action has succeeded and the process ends. It should be apparent to those skilled in the art that the flowchart in
The process shown in
To decrease the failure rate, an exemplary method 500 shown in the flowchart of
In an exemplary embodiment, if the “getting thread state” at process block 414 fails, performance counters are measured to determine when a UI element will be ready to perform an action. In other embodiments, performance counters are measured without performing the processes detailed in
To measure performance counters, two readings of certain performance counters (502) are taken with a time Δ between the two readings. Which performance counters to measure is implementation dependent, but generally, suitable performance counters are those associated with the memory and CPU consumption by the target process. Sample suitable performance counters are the CPU cycles for the target UI control, or some other measure of the process activity. Specific operating systems may provide suitable performance counters.
At process block 502, a first measure of the performance counters is taken. At process block 504, the process sleeps until some time has passed, some event occurs or, in some embodiments, it returns immediately. This process block is explained with more detail in
At process block 506, a second measure of the same performance counters is taken. At process block 508, it is determined if the two readings are essentially identical, as explained below. If so, then it is assumed that the target process is ready for user input. If the two performance counters' measurements are too far apart, then, at process block 510, the test program sleeps. As mentioned, a more complete explanation of the sleep process is given with reference to
At process block 502, the process checks the performance counters again. This process continues until either the two performance counter readings are essentially identical or until a user-defined time-out criterion is reached, discussed with reference to
After two essentially identical performance counter readings are found, at process block 512, the program again attempts to select the UI element.
To improve the performance to essentially 100%, an exemplary method 600 shown in the flowchart of
At process block 602, the UI element that is to have an action performed on it is located. At process block 604, it is determined if locating the UI element was successful. If not, then the process continues at process block 606, where the process sleeps for some period, and again tries to locate the element at process block 602. A more complete explanation of the sleep process is given with reference to
Once the UI element is successfully located, the desired action is performed on the UI element, as shown at process block 608. Often, this entails the test automation program selecting the UI element. At process block 610, it is determined if the action completed successfully. If the action did succeed, then the method has successfully completed. If the action did not succeed, then the process continues at process block 612, where the process sleeps, and again attempts to perform the action. In an exemplary embodiment, the process sleep time is 500 ms. In an exemplary embodiment this approach is combined with those depicted on
An exemplary method of the sleep process block mentioned with reference
At process block 702, it is determined if some user-defined criterion has been met, if so then the process continues at process block 704. If not, the process continues at process block 706. Process block 702 is generally used to determine if the process has been waiting for too long a period for the requested action to happen.
In an exemplary embodiment the user-defined criteria asks if the total time waiting for this specific action has been 3 minutes or longer, if waiting for a window response; and 2 minutes or longer, if waiting for a window element response. Another embodiment counts the number of times that it has entered the sleep module and then throws an exception when a maximum number has been reached. In other embodiments the criterion is a total wait time. In yet other embodiments the criterion is the occurrence of an event; other possible criteria should be obvious to someone of skill in the art.
At process block 704, an exception is thrown—the program should handle it accordingly. If the user-defined criterion has not been met, the process continues at process block 706 where the program pauses; a number of pause criteria are acceptable. In some instances, as shown in process block 708, the program sleeps for some designated period of time. Ideal sleep time is dependent upon the underlying hardware, the specific application, and exactly what is being waited for; but a basic rule of thumb is that if the wait time is too small, the computer system is not given enough time to change state or the test program will be using too many CPU resources by performing too many checks, but if the wait time is too long, it can degrade application running time. In an exemplary embodiment, each invocation of the sleep time has at least the potential of being different.
In process block 710, the process waits until a certain amount of time expires or until some event occurs, whichever happens first. In process block 712, the process returns to the calling program immediately. Each of the choices in process block 706 may be invoked within the same application at different places, a single application may invoke only a subset of the choices, or a different method of determining sleep time may be used. After the sleep of whatever sort, at process block 714, control returns to the calling process.
The implementations described here are technology agnostic, in that they should be able to be built into the underlying applications at a low-enough level that the implementation is invisible to users of the automatic testing programs; objects are selected without any awareness of the underlying synchronization.
Having described and illustrated the principles of the illustrated embodiments, it will be recognized that the various embodiments can be modified in arrangement and detail without departing from such principles. It should be understood that the programs, processes, or methods described herein are not related or limited to any particular type of computing environment, unless indicated otherwise. For example, the technologies described herein may be applied to any user interface function that needs synchronization with threads other than its own.
Various types of general purpose or specialized computing environments may be used with or perform operations in accordance with the teachings described herein. Elements of embodiments shown in software may be implemented in hardware and vice versa. Various types of general-purpose or specialized computers may be used to perform the systems and methods disclosed herein. Furthermore, computer-readable media comprising computer-executable instructions for performing such actions can achieve actions described herein.
In view of the many possible embodiments, it should be recognized that the detailed embodiments are illustrative only and should not be taken as limiting the scope of our invention. Rather, I claim as my invention all such embodiments as may come within the scope and spirit of the following claims and equivalents thereto.