Patent Application
20110239141
The present invention relates to the field of computer applications, and particularly, relates to a Graphical User Interface (GUI) element identification method for identifying a target GUI element on a GUI more accurately, as well as an apparatus and a system using the same.
Currently, there are a great amount of Graphical User Interfaces (GUIs) in computer applications. For various purposes, GUI element identification becomes an important research subject. However, some of GUI elements are difficult to be identified correctly due to complexity of the applications themselves and poor technical means. Thus, many problems may be caused in practices.
For example, it is often required in a practical application that a non-WEB application, i.e. a local application, may be migrated to a WEB-based application by the migration technique, in which a lot of identification of GUI elements for the local application is involved. A consistency between the two applications before and after migration in terms of GUI elements must be guaranteed. Otherwise, a consistency between the two applications before and after migration in terms of views and functionalities cannot be implemented. The inconsistency may cause failure of the migration and thus a disastrous result.
Currently, a GUI element is generally identified with attribute information of the GUI element. For example, this attribute information may be XML descriptions of the GUI element, i.e., XPath information of the GUI element. Reference 1 (CN101369249A) discloses a method of identifying a GUI element by means of XPath. Hereinafter, the XPath based identification method will be described with reference to
For this purpose, training may be performed firstly. During the training, all user interface (UI) information on the current GUI of the local application may be acquired by, e.g., a UI dump unit. In the example as shown in
During running, all UI information of a current GUI may be acquired by the UI dump unit. Here, it is assumed that the GUI presented during the running is still the GUI 1000 as shown in
Of course, respective XPath information for a plurality of target elements may be recorded during the training. Then, respective GUI elements in a current GUI may be identified respectively based on the recorded XPath information during the running.
However, generally some GUI element may not be uniquely identified by means of XPath, especially when same items exist in the XPath information as recorded for a plurality of GUI elements.
As seen from above, the GUI elements 3002 and 3003 are different. However, according to the XPath method, both of them are described as:
Thus, the GUI elements 3002 and 3003 can not be distinguished correctly by means of XPath.
Hence, there is a need for an improved element identification method, as well as an apparatus and a system using the same, to identify GUI elements correctly.
In view of the above, an object of the present invention is to provide an enhanced Graphical User Interface (GUI) element identification method, as well as an apparatus and a system using the same. It is possible to significantly improve the accuracy in identifying GUI elements by means of the identification method, the apparatus and the system, so as to correctly identify the GUI elements.
According to an aspect of the present invention, there is provided a method of generating GUI element identification information, comprising: displaying a GUI which includes a target GUI element; analyzing the displayed GUI, to acquire attribute information for the target GUI element; acquiring extra information, other than the attribute information, related to the target GUI element; and processing the attribute information and the extra information to generate the identification information for the target GUI element.
Preferably, the attribute information may comprise XPath information, and acquiring the attribute information may comprise: acquiring information, which is in XML format, on all elements within the GUI; and analyzing the element information in XML format to acquire the XPath information for the target GUI element.
Preferably, the extra information may comprise information on historical event(s) causing the target GUI element to occur, and acquiring the extra information may comprise: recording the historical event(s) causing the target GUI element to occur; and analyzing the historical event(s) to transform it/them into occurring condition(s) for the target GUI element, wherein the occurring condition(s) become(s) the information on historical event(s) for the target GUI element.
Preferably, the extra information may comprise system state information.
According to another aspect of the present invention, there is provided a method of identifying a GUI element to be identified based on GUI element identification information, the GUI element identification information comprising attribute information and extra information for respective GUI element(s). The method comprises: displaying a GUI which includes the GUI element to be identified; analyzing the displayed GUI, to acquire attribute information for the GUI element to be identified; acquiring extra information, other than the attribute information, related to the GUI element to be identified; and searching the GUI element identification information for an information item with attribute information matching the acquired attribute information for the GUI element to be identified and also with extra information matching the acquired extra information related to the GUI element to be identified.
Preferably, the extra information may comprise historical event information and/or system state information.
According to still another aspect of the present invention, there is provided an apparatus of generating GUI element identification information, comprising: a display device adapted to display a GUI which includes a target GUI element; an is attribute information acquiring unit adapted to analyze the displayed GUI to acquire attribute information for the target GUI element; an extra information acquiring unit adapted to acquire extra information, other than the attribute information, related to the target GUI element; and an identification information generating unit adapted to process the attribute information acquired by the attribute information acquiring unit and the extra information acquired by the extra information acquiring unit to generate the identification information for the target GUI element.
Preferably, the attribute information may comprise XPath information, and the attribute information acquiring unit may comprise: an UI dump unit adapted to acquire information, which is in XML format, on all elements within the GUI; and an attribute information generating unit adapted to analyze the element information in XML format acquired by the UI dump unit to acquire the XPath information for the target GUI element.
Preferably, the extra information may comprise information on historical event(s) causing the target GUI element to occur, and the extra information acquiring unit may comprise: an event recording unit adapted to record the historical event(s) causing the target GUI element to occur; a target GUI element determining unit adapted to determine whether the target GUI occurs on the GUI or not, and to send the determination result to the event recording unit; and an extra information generating unit adapted to generate the extra information comprising the information on the historical event(s) based on the historical event(s) for the target GUI element recorded by the event recording unit.
According to yet another aspect of the present invention, there is provided an apparatus of identifying a GUI element, comprising: storage means adapted to store GUI element identification information comprising attribute information and extra information for respective GUI element(s); a display device adapted to display a GUI which includes the GUI element to be identified; an attribute information acquiring unit adapted to analyze the displayed GUI to acquire attribute information for the GUI element to be identified; an extra information acquiring unit adapted to acquire extra information, other than the attribute information, related to the GUI element to be identified; and an identifying unit adapted to identify the GUI element to be identified based on the attribute information acquired by the attribute information acquiring unit and the extra information acquired by the extra information acquiring unit as well as the GUI element identification information stored in the storage means.
According to yet another aspect of the present invention, there is provided a GUI element identification system, comprising the apparatus of generating GUI element identification information according to the present invention and the GUI element identifying apparatus according to the present invention, wherein the storage means stores the GUI element identification information generated by the apparatus of generating GUI element identification information.
Preferably, the attribute information acquiring unit included in the apparatus of generating GUI element identification information and the attribute information acquiring unit included in the GUI element identifying apparatus are made by a same unit, and the extra information acquiring unit included in the apparatus of generating GUI element identification information and the extra information acquiring unit included in the GUI element identifying apparatus are made by a same unit.
According to the present invention, due to the introduction of the extra information into the identification of the elements, it is possible to overcome the defect that in some cases some GUI elements cannot be distinguished just based on the attribute information, and thus it is possible to greatly improve the accuracy in identifying GUI elements.
The above and other objects, features and advantages of the present invention will be more apparent by the following descriptions of embodiments of the present invention with reference to the drawings, in which:
Hereinafter, the present invention will be described in detail by referring to the embodiments as shown in the drawings. However, it should be understood that the descriptions are only Illustrative and are not intended to limit the present invention. Further, in the following, descriptions on well-known structures and techniques are omitted so as not to obscuring concepts of the present invention.
Some forms of GUIs are shown in the drawings as examples. However, it should be noted that these GUIs are only drawn illustratively and in practices there may be various GUIs in various forms. Further, some process steps and processing modules are shown in the drawings. However, some of these process steps/processing modules may be combined into a single step/module, or a single one of these process steps/processing modules may be divided into several sub-steps/sub-modules. It should be appreciated that these process steps/processing modules may be implemented in hardware, software, firmware or combinations thereof.
According to the present invention, identification information which is enhanced over the attribute information in the prior art is used for identifying GUI elements.
For this purpose, the present invention involves two aspects as follows: generation of the enhanced identification information (i.e. a training phase), and identification of the GUI elements based on the enhanced identification information (i.e. a running phase). Hereinafter, these two aspects will be described respectively referring to the drawings.
It should be noted that in the following descriptions “XPath information” is illustrated as an example of “attribute information”. However, the “attribute information” is not limited to the “XPath information”, and may comprise descriptions of GUI elements in other languages besides XML. For example, information as described in any of programming languages such as C, Basic and Python, or non-programming languages such as Text may be used as the attribute information.
Then, attribute information of the target GUI element, e.g. XPath information, may be collected in step S403, and extra information, other than the attribute information, related to the target GUI element may be collected in step S404, respectively. Here, the steps S403 and S404 may be performed in parallel or sequentially. Then in step S405, the attribute information and the extra information collected for the target GUI element may be processed to generate enhanced identification information for the target GUI element. Here for example, the attribute information and the extra information may be combined to form the enhanced identification information. Finally, the method ends in step S406.
Hereinafter, the steps S403 and S404 will be described in detail with reference to an example illustrated in
In the prior art, there are many approaches to acquire the attribute information of the GUI elements such as the XPath information. For example, in the step S403, information about all UI elements on the GUI (including the target GUI element 1 or 2) may be acquired, the information being formed in XML format. Finally, the UI element information in XML format, as shown in
Then, the acquired UI element information in XML format may be analyzed to obtain the XPath information of the target GUI element. Here, as discussed in the description with reference to
The characteristics of the present invention further consist in that the extra information related to the target GUI element may be acquired in step S404. For example, the “extra information” may comprise: information on historical event(s) resulting in occurrence of the target GUI element, e.g. operation events of a user by means of input devices such as keyboard and mouse; and/or system state information related to the target GUI element, e.g. occupied hardware (i.e. memory, CPU, etc.) resources or software resources (such as the number of GUI objects), and the like. In the following descriptions, the historical event information is taken as an example. However, it should be noted that the “extra information” in the present invention is not limited to such historical event information.
In step S404, in order to acquire the historical event information, the historical event(s) resulting in occurrence of the target GUI element may be recorded firstly. For example, the events which cause the occurrence of the target GUI element 1 (5002 in
After the historical event(s) resulting in the occurrence of the target GUI element has/have been recorded, the historical event(s) may be analyzed to obtain conditions under which the target GUI element will occur. Such analysis may be completed manually or by an automatic training program during the training phase. Here, for convenience of descriptions, the occurrence conditions for the respective GUI elements may be described in logic expressions as follows:
It should be noted that the above logic expressions are only given for convenience of the explanations. In practical applications, the occurrence conditions for the target GUI element may be described in various ways (for example, in form of tables). These occurrence conditions represent the historical event information which causes the occurrence of the target GUI element.
After the attribute information (here, the XPath information) and the extra information of the target GUI element are acquired as illustrated above, they are processed (e.g. combined) in step S405 to obtain the identification information for the target GUI element. For example, for the example shown in
After the enhanced identification information of the GUI element is acquired during the training phase as described above, the GUI element may be identified with the enhanced identification information during the running phase.
Then, attribute information, such as XPath information, of the GUI element to be identified may be collected in step S703, and extra information, other than the attribute information, related to the GUI element to be identified may be collected in step S704, respectively. The collections of the attribute information and the extra information are similar to those during the training phase, and thus detailed descriptions thereon are omitted here.
After the attribute information and the extra information for the GUI element to be identified are acquired, then in step S705, the GUI element identification information as recorded during the training phase is searched to find whether there is an item matching the acquired attribution information and extra information. If there is such a matching item, the GUI element to be identified is identified as one corresponding to the matching item in step S706. Otherwise, an error process, e.g. reporting to the user, may be performed in step S707. Finally, the method ends in step S708.
The above identification process will be described in detail in connection with the example as illustrated in
In a case where the GUI element 3001 exists on the GUI currently displayed by the local application, if the “OK” key is directly clicked without clicking the checkbox (as shown by the arrow 1 of
At the same time, the event which causes the occurrence of the GUI element 3002 is:
The above acquired XPath information and historical event information for the GUI element 3002 may be compared with the identification information (as shown in
Hence, it can be known from the comparison that the GUI element 3002 matches the identification information of the target GUI element 1 as illustrated in
Similarly, in a case where the GUI element 3001 exists on the GUI currently displayed by the local application, if the checkbox is checked by being clicked once and then the “OK” key is clicked (as shown by the arrow 3 of
At the same time, the event which causes the occurrence of the GUI element 3003 is:
The above acquired XPath information and historical event information for the GUI element 3003 may be compared with the identification information (as shown in
Hence, it can be known from the comparison that the GUI element 3003 matches the identification information of the target GUI element 2 as illustrated in
In this way, the GUI elements 3002 and 3003 which would not be distinguished in the prior art may be differentiated by the method of the present invention.
According to the present invention, there is further provided an apparatus of generating GUI element identification information.
The apparatus 800 may further comprise a storage device 804. For example, the storage device 804 may be used for storing the attribute information acquired by the attribute information acquiring unit 801, the extra information acquired by the extra information acquiring unit 802 and/or the identification information generated by the identification information generating unit 803 etc. The storage device 804 may be a memory inside the apparatus 800 of generating the identification information, or may be an external memory and even a network memory. The is storage device 804 may comprise a single memory, and may further comprise a plurality of memories.
Here, it is to be noted that the element determining unit 8022 is required here to determine whether the target UGI element occurs or not. This is because it is still in the training phase when the identification information are being generated and thus there must be “someone” (or an automatic training program) to notify the computer of the occurrence of the target GUI element. After the identification information of the GUI element is acquired during the training phase, the GUI element may be identified automatically during the running phase.
Here, the element determining unit 8022 may notify the event recording unit 8021 of the determination result as illustrated above. Optionally, the element determining unit 8022 may notify the attribute information acquiring unit 801 (specifically, the attribute information generating unit 8012, for example) of the result so that the attribute information acquiring unit 801 may acquire the attribute information for the target GUI element correctly.
According to the present invention, there is further provided an apparatus of identifying a GUI element.
Here, the GUI element identification information is stored in the storage means 1104. For example, the information may be acquired by performing the method as illustrated in
Further, the attribute information acquiring unit 1101 comprised in the apparatus 1100 of identifying the GUI element may have the same structure as the attribute information acquiring unit 801 comprised in the apparatus 800 of generating the GUI element identification information as described above. Also, the extra information acquiring unit 1102 comprised in the apparatus 1100 of identifying the GUI element may have a structure similar to the extra information acquiring unit 1102 comprised in the apparatus 800 of generating the GUI element identification information as described above, except that the target GUI element determining unit 8022 is not needed.
According to the present invention, there is further provided a GUI element identification system, comprising the above apparatus of generating GUI element identification information and the GUI element identifying apparatus according to the present invention. In the system, the generation of the GUI element identification information (the training phase) and the identification of the GUI element (the running phase) may be completed. In this system, the apparatus of generating the GUI element identification information and the GUI element identifying apparatus may be separated devices (e.g., two independent devices connected via the network). Or alternatively, the apparatus of generating the GUI element identification information and the GUI element identifying apparatus may be integrated into a single entity. Specifically, the respective attribute information acquiring units (801, 1101), extra information acquiring units (802, 1102) in the apparatus of generating the GUI element identification information and the GUI element identifying apparatus may be constituted by one same unit, or may be formed as separated units. The GUI element identification information generated by the apparatus of generating the GUI element identification information may be stored in the storage means of the GUI element identifying apparatus.
The present invention is illustrated in the above with reference to the embodiments thereof. However, those embodiments are just provided for illustrative purpose, and are not intended to limit the present invention. The scope of the present invention is defined by the following claims as well as equivalents thereof. Therefore, those skilled in the art can make various modifications and substitutions and thereto without departing from the scope of the present invention, which all fall into the scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201010137015.4 | Mar 2010 | CN | national |
