Patent Application
20050091510
The technical field relates to graphical user interface components of a computer program. More particularly, the field relates to computer implemented methods and systems for identifying elements or components of a graphical user interface system of a computer program.
A typical mechanism for implementing user interaction with a computer may involve the use of a graphical user interface (GUI) associated with a computer program running on the computer. Generally speaking, user interface elements (hereafter, UI elements) are aspects of a computer system or program which may be seen (or heard or otherwise perceived or interacted with) by an end user and used to control the operation of a computer system or program through commands and other mechanisms. A graphical user interface emphasizes the use of graphical UI elements which are responsive to user input (e.g., mouse events, keyboard data entry etc.) for interacting with a computer system or program. Common UI elements familiar to most users include, buttons (e.g., “OK” or “Cancel), edit boxes, list boxes, combo boxes, scroll bars, pick lists, pushbuttons, radio buttons and components of World Wide Web pages (e.g., hyper-links, documents and images).
In a typical computer program it is not uncommon to encounter literally thousands of such UI elements. Also, some of the same UI elements may appear multiple times as part of different composite UI elements depending on a current context of the program's use. Also, the same UI elements may appear in other computer programs as well. Nevertheless, each instance of a UI element may need to be uniquely identified for various reasons such as programming the functionality of the program hosting such UI elements, testing such a program or for use in otherwise identifying the particular UI element to another program module in communication with the host program. For instance, Assistive Technology (AT) software, like screen readers, screen enlargers or alternative input devices, as well as test automation programs that communicate with application programs may need to clearly identify UI elements hosted within the application programs. More particularly, in one potential scenario, suppose an application program tester has found a defect related to a UI element in an application program and wants to continue to exercise the application at the same UI element to determine whether the defect was an aberration or permanent. In order to do so, the tester may want to reboot his computer and get back to the same defective UI element for testing.
However, currently there is no reliable mechanism of identifying these UI elements across reboots, across instances of an application, across localized versions, and across builds. Currently, depending on the operating system platform, it is possible to use a combination of role, name, location, position among siblings (e.g., 4th child among n children), and other things. However, none of these properties on their own are sufficient; and many of them are fragile (e.g., location, name).
Thus, there is a need for systems and methods to provide identifiers for a UI element that persist across instances of an application, across localized versions, across builds, across reboots and other similar events. More particularly, there is a further need for platform level Application Program Interfaces (APIs) that can be used by a client program to receive persistent identifiers related to a source program and use such an identifier to locate a particular UI element in a target program.
Described herein are Application Programming Interfaces that enable client applications to request and receive identifiers for user interface elements in a graphical user interface of a computer program, wherein the identifiers are capable of persistently identifying the user interface elements across different instances of the computer program, across different builds of the program, across reboots of the computer running the computer program etc. A persistent identifier is one that is not fragile and maintains its identity across events such those mentioned above.
In one aspect, the persistent identifiers are generated by using a hierarchical tree structure representation of the graphical user interface comprising the user interface element of interest. The identifiers are made persistent by including in the identifier, identifying information not only of the user interface of interest but also as much of the identifying information as is appropriate of the parent elements above the user interface of interest in the hierarchy of the tree structure. Such information describes a hierarchical path of inheritance for the user interface of interest within the graphical user interface. The path may be referred to as an element path and the identifier comprising a description of the path may be referred to as an element path identifier.
Furthermore, a set of Application Programming Interfaces are described herein that enable a client program to locate a target user interface in a graphical user interface of a target computer program by using the element path identifier of the target element. For example, by using APIs for generating persistent identifiers a client program can mark a user interface of interest and later use the persistent identifier to locate the user interface of interest in another instance of the program, across another build of the program and across other events that would have rendered any of the fragile identifiers useless.
In another aspect, a tree structure representation of a graphical user interface may comprise at least one branch portion that is associated with a strong name and has a scope inside of which elements of the strongly named branch are guaranteed to be uniquely identifiable by using a named branch element identifier. Thus in one embodiment of generating an element path identifier, elements of a named branch portion may be uniquely identified by their named branch element identifier alone. In yet another embodiment, sections of an element path comprising strongly named branches may be identified by the strong name of the named branch and the named branch element identifier of the lowest element still within the scope of the named branch portion. This reduces the quantity of identifying information stored in a elment path identifier data structure. However, the efficiency of searches for target user interface elements can be improved by adding information related to all the elements in named branch portion.
The persistent identifier can be further improved by adding additional information within an element path identifier for reducing any residual ambiguity. For instance, sibling order information can focus a search and reduce ambiguity.
Additional features and advantages of the systems and methods described herein will be made apparent from the following detailed description that proceeds with reference to the accompanying drawings.
Although individual elements of a graphical user interface (e.g., a radio button) may appear to the user as a single composite item, it may actually be represented in the computer as a number of separate items or sub-elements that have been combined together. Furthermore, each of these sub-elements themselves can be composed from other sub-elements. In this manner, UI elements can serve as building blocks for building other, more complex, UI elements. Such an approach is useful because the software managing the user interface (e.g., the user interface framework) can re-use the definitions of certain common elements when assembling them into composite elements.
Any one of the elements of the user interface illustrated in
Although it is possible to identify UI elements using identifiers such as those listed above, none of them can singularly provide a strong identifier that is likely to persist across a reboot of the computer running the program, across a different build of the program when still in development, across the opening of an another instance of the same program, or opening of the same type of program on another computer. In such situations these identifiers alone prove to be fragile or too ambiguous. For instance, control identifiers (control IDs in Microsoft® Windows) by themselves are not very unique. On a given Microsoft® Windows desktop, for instance, two different UI elements (e.g., a button and a dialog box) running on two different applications may be assigned the very same control ID. Thus, in this example, searching by a control ID alone cannot unambiguously identify a selected one of those UI elements.
However, combining the control ID with another identifier may be helpful. Take for instance the same example as above, even if the control ID for the button element and the dialog element were same, if a composite identifier combined the control ID and the control class name associated with the individual UI element then the button UI element and the dialog box UI element may distinguishable from each other. Similarly, generating composite identifiers having additional information capable of identifying a UI element can increase the strength of the identifier's uniqueness and reduce the possibility of ambiguity.
Adding as much identification information that is directly associated with a UI element into a composite identifier of a UI element may be helpful in identifying it unambiguously. However, given the countless number of UI elements and the complexities involved, it is may not succeed all the time. Thus, yet another improvement may be to also add identification information related to parent elements to the composite identifier of a UI element. For instance, a UI element path description identifier can be provided which comprises not only identification information related to a target UI element for which a unique persistent identifier is to be generated but it may also include information related to its parent elements and possibly even its children and sibling elements.
An element path description identifier may be able to serve as a strong identifier for a UI element because, even if the target UI element has other identifiers (e.g., module name, control ID, control class name etc.) that are identical to another UI element, it is less likely that they both would have the same parent elements, also with identical identifying information. Through concept of a path related identifier a UI element's unique hierarchy and parentage can be leveraged to identify it uniquely and persistently.
Persistent identifying information related to a selected source UI element may be recorded and later used as a parameter for a method of searching and locating a target UI element that matches the identifier of the source UI element. For example, the source element and target element may be different instances of the same element. For instance, a button of interest in a word processor program may be given a persistent identifier and the identifier may be used to locate the same button on the same program being run on a different computer. As shown in
According to one embodiment, the method of recording or generating an element path description identifier for a source UI element may be implemented in a software component as described in
The information recorded at each UI element in an element path (e.g., 210) may be a function of what is exposed in that particular UI element's object model and may also depend on the user interface platform used to create the UI element. For example, it is possible for a user interface to comprise UI elements from a number of different UI platforms (e.g., Win 32 by Microsoft® Corporation, and Swing for Java etc.). Thus, the method of recordation should be capable of identifying a UI element as belonging to a certain user interface platform and record its identifying information accordingly. The actual information about each UI element recorded may include such items as control ID, module identifier, control class name etc. Initially, as the method is progressing up a tree representation, some identifying information (e.g., module identifier for identifying a UI element 205 as belonging within a certain application module 215B) may not be exposed until a higher element is reached. In that event, such information is recorded upon being exposed.
In the previous example, the topmost node in the user interface hierarchy 215A was also designated as the element path root node where the recording of the identifier information for the element path identifier was to be stopped. However, it is possible to designate a lower element (e.g., the application node 215B or the frame node 215C) to be designated as the element path root. This may be useful where a particular client system can ensure strong and persistent identification of elements up to a certain level in the user interface hierarchy or can devise other local methods of searching and identification of UI elements at certain levels. The element path root node may be provided as a parameter to the process 510.
Once an element path identifier is recorded, for instance, it may be used as a parameter in a search method applied to a target application to locate a UI element matching the identifier within the application. Such a process may be implemented in a software component for searching UI elements as described in
According to one embodiment, the method 420 of locating the target UI element upon receiving a specification of an element path identifier may be implemented in a software component as described in
The search process 600 described above prunes the tree structure 200 by progressing along the element path 210 one level at a time wherein one of the sibling elements forming a separate branch is distinguishable over the others of its siblings. Such a process eliminates the need to search every branch in a tree structure. However, it is also possible to search all branches from a certain branch root of the tree structure if identifying information that distinguishes a branch root node from its siblings is not available in the element path information. For example, the branch root node 215D may not be distinguishable from the branch root node 216 based on the element path information. However, such a search process may be more complex and would certainly be more time and resource consuming than the process described with reference to
The methods of generating element path identifiers for selected source UI elements and the complementary process of identifying or locating a target UI element according to such an identifier may be implemented within a system as shown in FIG. 7. The system may comprise an element path engine 710 programmed for generating element path identifiers of source UI elements hosted within a source program 730 upon receiving a request to do so via a client program 720. Furthermore, the element path engine 710 may also locate or identify a target UI element within a target program 715 based on an element path identifier provided via a client program 720. The communication between client program 720 and the element path engine 710 may be enabled via an Application Programming Interface (API) 725 which can allow the client program to pass function calls including element path description identifiers as its parameters to functions implemented for searching of target UI elements. Also, the API 725 may be further capable of passing function calls including specification of one or more source UI elements as parameters to functions implemented for generating element path identifiers of specified UI elements. Embodiments of such APIs will be described in further detail below.
The client program 720 may be a test program which is exercising an instance of a program (e.g., 730) such that the test program may identify a source UI element of interest for which it may request a persistent identifier such as its element path identifier through the element path engine 710. Later on, the test program (e.g., the client 720) may want to use the persistent identifier to locate another instance of the same UI element in another program (e.g., the target program 715). The source program 730 and the target program 715 may be located on the same or different computers.
In yet another scenario, the client program 720 may use the element path engine 710 to record a user's access of various UI elements within a source program 730 by identifying the accessed UI elements using the persistent identifiers and later play back the user's access scenario on another program (e.g., 715). For example, such an implementation may be helpful in creating a help menu where users are walked through the procedures of using a program by the way of an example. The above examples are meant to be illustrative and other uses for a persistent identification of UI elements are possible.
As noted above, identifiers of UI elements within an element path can comprise a wide range in the strength of their identity with a particular UI element. Thus, a process of recording or generating an element path or even the process of searching for a UI element according to its element path can be adjusted depending on the strength of the identifier. For instance, suppose the branch representation 1010 of
The strong name associated with a named branch 1020 may be a file name or any other type of strong identifier of the named branch root 1021. For example, such a name may be referred to as a named branch root identifier and each UI element within a scope related to the named branch root may be uniquely identifiable by a named branch element identifier (hereafter, named branch element ID). An element path identifier data structure for a user interface comprising such a named branch may need additional fields for identifying UI elements (e.g., 1022) that belong within the scope of named branch 1020. For instance,
The user interface tree structure representation 1010 comprising a named branch portion 1020 may have an element path corresponding to a UI element 1025 as shown in
Thus, as shown in
Strictly recording only the minimum amount of identification information that is necessary for locating a UI element may not always be helpful for increasing the speed of a search. The named branch 1320 comprises relatively few sub-branches (two) and a small number of sub-elements. This may not always be true. Searching for a sub-element by its element path identifiers which do not contain the identification information for all its parent elements may mean all branches have to be exhaustively searched which can be time and resource consuming. Thus, including (in an element path identifier) the identification information of parent elements of a named branch UI element (as shown in
This can also be helpful in other circumstances. For instance,
However again in this scenario, as before, recording the identifier information regarding parent elements 1322 and 1324 of a UI element of interest 1325 may help improve the efficiency and speed with which it is located later. For instance, as shown in
Furthermore, the process of recording an element path description and the process of searching for target UI elements can adaptively record information based on the platform affinity of the UI elements on a given path.
The method of recording an element path identifier for a source UI element (e.g., the methods of
The API for generating or recording an element path identifier for a source UI element may be as follows in code representation:
The API for generating an element path description identifier for a source UI element may also be implemented as follows in code representation:
Complementing the above two APIs may be APIs for searching or locating target UI elements based on their respective element path descriptions. Such APIs may be as follows:
A find method can also be implemented that receives a string type description of an element path as an input parameter. Such an implementation may be as follows:
These APIs may be further refined by adding other parameters. For instance, GetElementPath API can have a flag parameter that could be set such that sibling order of each UI element in an element path hierarchy is recorded as part of the ElementPath result when the flag is set to be true. Then later in the FindLogicalElement the sibling order of UI elements in an element path may be useful for resolving ambiguity. The same may apply to the GetElementPathString and FindLogicalElementString APIs Other specific implementations of the APIs are possible that implement the functionality described above without the exact same naming convention for the methods and class names.
Other improvements are possible. For instance, if the element path root node need not be the topmost node in a hierarchy (e.g., 215A) then identifier information of whatever is the element path root node may be provided as a parameter of the GetElementPath API so that only information regarding the UI element of interest and the element path root node are recorded. Also, FindLogicalElement can have as a parameter the element path root node so that the search from a UI element is begun at the designated element path root.
Other specific implementations of the APIs are possible that implement the functionality described above without the exact same naming convention for the methods and class names. In fact, the implementations need not even be in an object-oriented language. In fact, the implementations need not even be in an object-oriented language.
Having described and illustrated the principles of our invention with reference to the described embodiments, it will be recognized that the described embodiments can be modified in arrangement and detail without departing from such principles. Although, the technologies described herein have been illustrated via examples of elements of graphical user interfaces, any of the technologies can use other user interface elements (e.g., sound). The APIs described herein have been illustrated with selected input parameters for the purposes of describing functionality of generating identifiers and searching by identifiers. Other parameters may be added or some of the parameters may be removed and still implement the principles of the invention. The same applies to the output parameters. Also, the element path identifiers have been described using selected component identifiers of elements in an element path. Other identifiers as appropriate for a selected user interface environment may be used as well. Furthermore, the functionality described herein with respect to the generation of identifiers and their use in searching and locating interface elements may be implemented across multiple software component modules. For instance, another software component may be introduced between the API modules and the client program or the element path engine to pass function calls, receive results and conduct other comparable actions.
Also, it should be understood that the programs, processes, or methods described herein are not related or limited to any particular type of computer apparatus. Various types of general purpose or specialized computer apparatus may be used with or perform operations in accordance with the teachings described herein. Actions described herein can be achieved by computer-readable media comprising computer-executable instructions for performing such actions. Elements of the illustrated embodiment shown in software may be implemented in hardware and vice versa. In view of the many possible embodiments to which the principles of our invention may be applied, it should be recognized that the detailed embodiments are illustrative only and should not be taken as limiting the scope of our invention. Rather, we claim as our invention all such embodiments as may come within the scope and spirit of the following claims and equivalents thereto.
