Web pages are documents on the World Wide Web that can be located, retrieved, and displayed in web browsers running on computers and/or mobile devices. Such web pages are written in the hypertext markup language (HTML), and have user interface (UI) presentations described using the cascading style sheet (CSS) language, which is typically used to describe the look-and-feel of web pages written in HTML. The CSS language enables webpage content to be separated from descriptions of UI presentations, which include information relating to UI elements such as images, video, text, etc. Using the CSS language, complex web pages are formatted on the fly, allowing the look-and-feel of the web pages to vary while the webpage content remains unchanged.
Web pages that have user interface (UI) presentations described using the cascading style sheet (CSS) language can pose particular challenges to software engineers, particularly those working in test environments. Whereas early webpage designs typically included arrangements of UI elements statically encoded in the webpage, today's complex webpage designs can include multilayered navigable collections of UI elements (e.g., images, video, text, etc.), which can be both statically and dynamically determined. As such, the detection of UI layout issues (e.g., overlap issues, truncation issues, etc.) in today's complex webpage designs has become increasingly difficult, often resulting in missed issues and/or false positive indications of issues.
Techniques are disclosed herein for detecting user interface (UI) layout issues for web applications. The disclosed techniques can include loading a target webpage to a web browser running on a client computer, injecting a detection script into code of the target webpage, and executing the detection script to analyze and/or detect one or more layout issues (e.g., overlap issues, truncation issues, etc.) pertaining to UI elements in the target webpage. The disclosed techniques can further include performing, by execution of the detection script, a recursive traversal of nodes of a document object model (DOM) tree corresponding to the target webpage. Based on the type of UI layout issues being analyzed and/or detected by the detection script, the recursive traversal of the DOM tree nodes can encompass all levels of the DOM tree or be limited to less than all levels of the DOM tree. Having performed the recursive traversal of the DOM tree nodes, the disclosed techniques can further include generating a suspect element list, which can list one or more UI elements in the target webpage for which a layout issue was detected. Once the suspect element list is generated, the disclosed techniques can further include performing a filtering process to remove false positive indications of UI layout issues from the suspect element list. Having removed the false positive indications from the suspect element list, the disclosed techniques can further include generating a client report of UI layout issues for the target webpage. By recursively traversing all levels of the DOM tree of the target webpage or less than all levels of the DOM tree based on the type of UI layout issues being analyzed and/or detected by the detection script, the disclosed techniques can advantageously reduce processing resource requirements while decreasing the occurrence of missed UI layout issues in the target webpage.
In one embodiment, a method of detecting user interface (UI) layout issues for web applications includes loading a target webpage to a web browser running on a client computer, and injecting a detection script into code of the target webpage. The method further includes executing the detection script to perform analysis and/or detection of UI layout issues pertaining to UI elements in the target webpage, in which the executing of the detection script includes (i) performing a recursive traversal of nodes of a document object model (DOM) of the target webpage, in which the nodes include at least structure information for rendering the respective UI elements in the target webpage, and, (ii) while performing the recursive traversal of nodes of the DOM, analyzing the structure information for one or more pairs of UI elements in the target webpage to detect one or more UI layout issues pertaining to the pairs of UI elements in the target webpage. The method further includes, having detected the UI layout issues pertaining to the pairs of UI elements, automatically generating, in a client report, one or more suggestions for modifying at least some of the structure information to address one or more of the UI layout issues, and terminating the recursive traversal of nodes of the DOM of the target webpage.
In certain arrangements, the UI layout issues include one or more overlap layout issues, and the analyzing of the structure information for one or more pairs of UI elements in the target webpage includes analyzing the structure information included in one or more nodes at an intermediate level of the DOM to detect the overlap layout issues pertaining to the pairs of UI elements in the target webpage.
In certain arrangements, the pairs of UI elements include a respective pair having a first UI element and a second UI element, and the overlap layout issues include a respective overlap layout issue pertaining to the first UI element overlapping and completely covering the second UI element. In addition, the terminating of the recursive traversal of nodes of the DOM of the target webpage includes, having detected the respective overlap layout issue pertaining to the first UI element overlapping and completely covering the second UI element, terminating the recursive traversal of nodes of the DOM for detecting UI layout issues pertaining to the first and second UI elements at the intermediate level of the DOM, due to the respective overlap layout issue not being problematic in the target webpage.
In certain arrangements, the pairs of UI elements include a respective pair having a first UI element and a second UI element, and the overlap layout issues include a respective overlap layout issue pertaining to the first UI element overlapping and partially covering the second UI element. The method further includes, having detected the respective overlap layout issue pertaining to the first UI element overlapping and partially covering the second UI element, continuing the recursive traversal of nodes of the DOM for detecting UI layout issues pertaining to the first and second UI elements to at least another level beyond the intermediate level of the DOM.
In certain arrangements, the terminating of the recursive traversal of nodes of the DOM of the target webpage includes, having detected the respective overlap layout issue pertaining to the first UI element overlapping and partially covering the second UI element, terminating the recursive traversal of nodes of the DOM for detecting UI layout issues pertaining to the first and second UI elements at the other level beyond the intermediate level of the DOM.
In certain arrangements, the UI layout issues include one or more truncation layout issues, and the analyzing of the structure information for one or more pairs of UI elements in the target webpage includes analyzing the structure information included in one or more nodes at an intermediate level of the DOM to detect the truncation layout issues pertaining to the pairs of UI elements in the target webpage.
In certain arrangements, the pairs of UI elements include a respective pair having a first UI element and a second UI element, the second UI element is a container element that contains the first UI element, and the truncation layout issues include a respective truncation layout issue pertaining to (i) a first width of the first UI element being longer than a second width of the second UI element, and (ii) a portion of the first UT element being hidden from view within the second UI element. The method further includes, having detected the respective truncation layout issue pertaining to the first UI element and the second UI element, continuing the recursive traversal of nodes of the DOM for detecting UI layout issues pertaining to the first and second UI elements to at least another level beyond the intermediate level of the DOM.
In certain arrangements, the terminating of the recursive traversal of nodes of the DOM of the target webpage includes, having detected the respective truncation layout issue pertaining to the first UI element and the second UI element, terminating the recursive traversal of nodes of the DOM for detecting UI layout issues pertaining to the first and second UI elements at the other level beyond the intermediate level of the DOM.
In a further embodiment, a system for detecting user interface (UI) layout issues for web applications includes a memory, and processing circuitry configured to execute instructions out of the memory (i) to load a target webpage to a web browser, (ii) to inject a detection script into code of the target webpage, (iii) to execute the detection script to perform analysis and/or detection of UI layout issues pertaining to UI elements in the target webpage, in which the execution of the detection script includes (a) performing a recursive traversal of nodes of a document object model (DOM) of the target webpage, the nodes including at least structure information for rendering the respective UI elements in the target webpage, and, (b) while performing the recursive traversal of nodes of the DOM, analyzing the structure information for one or more pairs of UI elements in the target webpage to detect one or more UI layout issues pertaining to the pairs of UI elements in the target webpage, and, (iv) having detected the UI layout issues pertaining to the pairs of UI elements, to automatically generate, in a client report, one or more suggestions for modifying at least some of the structure information to address one or more of the UI layout issues, and to terminate the recursive traversal of nodes of the DOM of the target webpage.
In another embodiment, a computer program product includes non-transitory, computer-readable media having instructions that, when executed by processing circuitry of a computerized apparatus, cause the processing circuitry to perform a method of detecting user interface (UI) layout issues for web applications. The method includes (i) loading a target webpage to a web browser running on a client computer, (ii) injecting a detection script into code of the target webpage, (iii) executing the detection script to perform analysis and/or detection of UI layout issues pertaining to UI elements in the target webpage, including (a) performing a recursive traversal of nodes of a document object model (DOM) of the target webpage, in which the nodes include at least structure information for rendering the respective UI elements in the target webpage, and, (b) while performing the recursive traversal of nodes of the DOM, analyzing the structure information for one or more pairs of UI elements in the target webpage to detect one or more UI layout issues pertaining to the pairs of UI elements in the target webpage, and, (iv) having detected the UI layout issues pertaining to the pairs of UI elements, automatically generating, in a client report, one or more suggestions for modifying at least some of the structure information to address one or more of the UI layout issues, and terminating the recursive traversal of nodes of the DOM of the target webpage.
Other functions and aspects of the claimed features of this disclosure will be evident from the Detailed Description that follows.
The foregoing and other objects, features, and advantages will be apparent from the following description of particular embodiments of the disclosure, as illustrated in the accompanying drawings, in which like reference characters refer to the same parts throughout the different views.
Techniques are disclosed herein for detecting user interface (UI) layout issues for web applications. The disclosed techniques can include injecting a detection script into code of a target webpage, and running the detection script to analyze and/or detect one or more layout issues pertaining to UI elements in the target webpage. The disclosed techniques can further include performing a recursive traversal of nodes of a document object model (DOM) tree corresponding to the target webpage. Based on the type of UI layout issues being analyzed and/or detected by the detection script, the traversal of the DOM tree nodes can encompass all levels of the DOM tree or be limited to less than all levels of the DOM tree. By recursively traversing all levels of the DOM tree of the target webpage or less than all levels of the DOM tree based on the type of UI layout issues being analyzed and/or detected by the detection script, the disclosed techniques can reduce processing resource requirements while decreasing the occurrence of missed UI layout issues in the target webpage.
The memory 118 included in the client computer 104 can be configured with volatile storage (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) and/or non-volatile storage (e.g., magnetic memory, flash memory). The memory 118 can be further configured to store a variety of software constructs, including code/data for detecting webpage user interface (UI) layout issues 120, a web browser application 122 for executing a web application 124 and a specified test framework 124 for web applications, as well as an operating system, a document object model (DOM) tree generator, and/or any other suitable software construct. For example, the web application 124 may be any suitable computer-based software application or webpage (e.g., a webpage 110) hosted by the respective web server(s) 102.1, . . . , 102.n and accessible to the client computer 104 over the network(s) 108. Further, the test framework 126 may be any suitable test framework (e.g., a test framework 112) accessible to the client computer 104 over the network(s) 108 from the server computer 106, such as the Selenium® test framework for web applications.
The processing circuitry 116 included in the client computer 104 can be configured to operate in accordance with the various software constructs (e.g., the webpage UI layout issue detection code/data 120, the web browser application 122) stored in the memory 118. The processing circuitry 116 can be implemented in a variety of ways, using one or more processors running specialized software, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more discrete components, digital circuits, analog circuits, and so on, or any suitable combination thereof. The display 117 can be configured as any suitable computer display capable of visually displaying, to a user, complex web pages designed to include multilayered navigable collections of UI elements (e.g., images, video, text, etc.) statically and/or dynamically determined in the respective web pages.
Once the Administrators function of the webpage 200 is selected by the user, a number of additional UI elements can appear for display on the webpage 200, including a drop-down menu 206, a box 208 containing a hyperlink or text, and a button 210 for performing a particular function or act, in which each of the hyperlink/text contained in the box 208 and the function/act performable via the button 210 corresponds to a user selection made from the drop-down menu 206. For example, the drop-down menu 206 may be employed by the user to select an identity or identifier of an administrator entity (“Identity of entity”), the box 208 may contain a hyperlink or text pertaining to an e-mail address of the selected administrator entity (“E-mail address of en . . . ”), and the button 210 may be actuated by the user (e.g., by using a mouse, a trackpad, a keyboard) to invite the selected administrator entity to join a business network or other suitable network of administrator entities.
As shown in
The techniques disclosed herein for detecting user interface (UI) layout issues for web applications can include loading the webpage 200 to the web browser application 122 (also referred to herein as the “browser”) running on the client computer 104, injecting a detection script into code of the webpage 200, and executing (or interpreting) the detection script to analyze and/or detect one or more layout issues (e.g., overlap issues, truncation issues, etc.) pertaining to UI elements in the webpage 200. The disclosed techniques can further include performing, by execution (or interpretation) of the detection script, a recursive traversal of nodes of a document object model (DOM) tree corresponding to the webpage 200. Based on the type of UI layout issues being analyzed and/or detected by the detection script, the recursive traversal of the DOM tree nodes can encompass all levels of the DOM tree or be limited to less than all levels of the DOM tree. Having performed the recursive traversal of the DOM tree nodes, the disclosed techniques can further include generating a suspect element list, which can list one or more UI elements in the webpage 200 for which a layout issue was detected. Once the suspect element list is generated, the disclosed techniques can further include performing a filtering process to remove false positive indications of UI layout issues from the suspect element list. Having removed the false positive indications from the suspect element list, the disclosed techniques can further include generating a client report of UI layout issues for the webpage 200. By recursively traversing all levels of the DOM tree of the webpage 200 or less than all levels of the DOM tree based on the type of UI layout issues being analyzed and/or detected by the detection script, the disclosed techniques can advantageously reduce processing resource requirements while decreasing the occurrence of missed UI layout issues in the webpage 200.
The disclosed techniques for detecting UI layout issues for web applications will be further understood with reference to the following illustrative example, as well as
In preparation for performing functions or acts pertaining to the detection phase 240 (see
It is noted that a document object model (DOM) is a cross-platform and language-independent convention for representing and/or interacting with objects of a webpage written in the hypertext markup language (HTML) or any other suitable markup language. Further, elements of a DOM, which correspond to individual components of the particular markup language used to write the webpage, can be represented as nodes of a DOM tree. For example, the DOM tree generator running on the operating system of the client computer 104 may generate a DOM tree 203 from objects of the webpage 200, as illustrated in
Having loaded the webpage 200 and the test framework 224 to the browser 122, the client computer 104 performs the functions or acts pertaining to the detection phase 240 (see
Once the detection script 228 is injected into the code of the webpage 200, the detection script 228 is executed to detect the layout issues (e.g., the truncation and overlap issues) pertaining to the UI elements in the webpage 200, as indicated by reference numerals 220, 222. To that end, the detection script 228 performs a recursive traversal of nodes of the DOM tree 203, encompassing either all levels of the DOM tree 203 or less than all levels of the DOM tree 203 based on the type of UI layout issues being detected by the detection script 228. For example, the detection script 228 may be executed to detect the “overlap” type of UI layout issues pertaining to the UI elements in the webpage 200. As shown in
As described herein, to detect overlap issues in the webpage 200, the detection script 228 can perform a recursive (e.g., top-down, bottom-up) traversal of the nodes of the DOM tree 203, traversing top-down starting from the root node 0 (or one of the parent nodes 1.1, 1.2, . . . , m.1, . . . , m.n) to the leaf nodes 1, 2, n, . . . , n+1, and traversing bottom-up from the leaf nodes 1, 2, . . . , n, n+1 back toward the root node 0 (or the parent node 1.1, 1.2, . . . , m.1, . . . , or m.n), taking into account both the left sub-tree 246 and the right sub-tree 248 of the DOM tree 203. During the top-down traversal of the DOM tree 203, the detection script 228 extracts or otherwise obtains at least the structure information (e.g., positions/dimensions of UI elements) from the parent nodes 0, 1.1, 1.2, . . . , m.1, . . . , m.n in the respective Levels 0-m, and stores the structure information for each parent node in a candidate list. During the bottom-up traversal of the DOM tree 203, the detection script 228 can determine, by analyzing (e.g., comparing) structure information (e.g., positions/dimensions) for pairs of UI elements from the candidate list, whether a UI element can overlap and either completely or partially cover another UI element in the webpage 200.
During the bottom-up traversal of the DOM tree 203, once the detection script 228 determines, at an intermediate level 1, 2, . . . , or m of the DOM tree 203, that a UI element described by the structure information of a parent node overlaps and completely covers a UI element(s) described by the structure information of its child node(s), the bottom-up traversal of the DOM tree 203 can terminate at that intermediate level 1, 2, . . . , or m before reaching the root node 0, due to the structures of the respective elements not being indicative of an overlap layout issue for the webpage 200. Otherwise, if the detection script 228 determines, at an intermediate level 1, 2, . . . , or m of the DOM tree 203, that a UI element (such as the UI element 216; see
In addition to detecting the “overlap” type of UI layout issues, the detection script 228 may be further executed to detect the “truncation” type of UI layout issues pertaining to the UI elements in the webpage 200. To detect such truncation issues in the webpage 200, the detection script 228 can perform the recursive (e.g., top-down, bottom-up) traversal of the nodes of the DOM tree 203, as described herein with reference to the overlap layout issues. However, during the bottom-up traversal of the DOM tree 203, the detection script 228 can determine, based on the structure information obtained from the parent nodes 0, 1.1, 1.2, . . . , m.1, . . . , m.n in the respective Levels 0-m, (i) whether a UI element (e.g., a hyperlink or text) has a width that is longer than the width of another UI element (e.g., a box) that contains the hyperlink or text, and (ii) whether a portion of the hyperlink or text is hidden from view within the box. One such truncation issue (as indicated by reference numeral 220) involves an image of the box 208, which contains an image of the hyperlink or text pertaining to an e-mail address of the selected administrator entity. As shown in
Having generated the suspect element lists for overlap and truncation layout issues, the client computer 104 (or a remote computer) performs functions or acts pertaining to the filtering phase 242 (see
In this example, the suspect element list for overlap layout issues lists the UI elements 216, 218 as having possible overlap layout issues in the webpage 200. To determine whether or not the UI elements 216, 218 correspond to false positive indications of overlap layout issues, the function of generating screenshots 232 can be performed, by the client computer 104 (or a remote computer), to generate a screenshot image of each of the UI elements 216, 218 (see
If it is determined that the common region 223 corresponds to a transparent or blank image area, then the UI elements 216, 218 may be deemed to be false positive indications of overlap layout issues, because, in this case, the common region 223 would likely not obscure any visible (i.e., non-transparent or non-blank) portion(s) of the UI elements 216, 218 in the webpage 200. However, if it is determined that the common region 223 does not correspond to a transparent or blank image area, then the UI elements 216, 218 may not be deemed to be false positive indications of overlap layout issues, because the common region 223 would likely obscure a visible (i.e., non-transparent or non-blank) portion(s) of the UI elements 216, 218 in the webpage 200.
In this example, the suspect element list for truncation layout issues lists the UI element 208 (i.e., the box) and the hyperlink or text element (“E-mail address of en . . . ”) contained in the box 208 as having possible truncation layout issues in the webpage 200. To determine whether or not the box 208 and the hyperlink or text element contained in the box 208 correspond to false positive indications of truncation layout issues, the client computer 104 (or a remote computer) may modify the style and/or format attributes contained in a CSS file for the box 208 to make the image of the box 208 transparent (i.e., not visible) in the webpage 200. Having made the image of the box 208 transparent (i.e., not visible), the function of generating screenshots 232 can be performed, by the client computer 104 (or a remote computer), to generate a first screenshot image of the hyperlink or text element (“E-mail address of en . . . ”) in the webpage 200, using the structure information (e.g., position/dimension) for the hyperlink or text element maintained in the suspect element list for truncation layout issues. The client computer 104 (or a remote computer) may then modify the style and/or format attributes for the box 208 to make the image of the box 208 visible (i.e., non-transparent or non-blank) in the webpage 200, and generate a second screenshot image of the box 208 and the hyperlink or text element (“E-mail address of en . . . ”) contained in the box 208, using the structure information (e.g., positions/dimensions) for the box 208 and the hyperlink or text element maintained in the suspect element list for truncation layout issues. Having generated the first screenshot image and the second screenshot image, the function of image processing 234 can be performed, by the client computer 104 (or a remote computer), to analyze (e.g., compare) the first and second screenshot images to determine whether or not the two screenshot images are identical (or the same).
If it is determined that the first and second screenshot images are identical (or the same), then the box 208 and the hyperlink or text element contained in the box 208 may be deemed to be false positive indications of truncation layout issues, because, in this case, a visible (i.e., non-transparent or non-blank) portion of the hyperlink or text would likely not be hidden from view within the box 208 in the webpage 200. However, if it is determined that the first and second screenshot images are not identical (or not the same), then the box 208 and the hyperlink or text element contained in the box 208 may not be deemed to be false positive indications of truncation layout issues, because a visible (i.e., non-transparent or non-blank) portion of the hyperlink or text would likely be hidden from view within the box 208 in the webpage 200.
Once the UI elements deemed to be false positive indications of overlap or truncation layout issues have been identified, the function of issuing a report 236 of UI layout issues for the webpage 200 can be performed, by the client computer 104 (or a remote computer), to remove such UI elements from the suspect element lists for overlap/truncation layout issues, and to include any UI elements remaining in the suspect element lists for overlap/truncation layout issues in the client report 238 (see
An exemplary method of detecting UI layout issues for web applications is described below with reference to
Having described the foregoing illustrative embodiments, other alternative embodiments and/or variations may be made and/or practiced. For example, it was described herein that a common region of two UI elements in a webpage, in which one UI element overlaps and partially covers another UI element, may not be deemed to be a false positive indication of an overlap issue if the common region does not correspond to a transparent or blank image area. In such a case, the common region would likely obscure a visible (i.e., non-transparent or non-blank) portion of one or more of the UI elements. In one embodiment, having determined that such a common region of partially overlapping UI elements in a webpage does not correspond to a transparent or blank image area, the disclosed techniques can automatically relocate at least one of the UI elements in the webpage so as to avoid obscuring a visible (i.e., non-transparent or non-blank) portion of the respective UI elements.
For example, the disclosed techniques may determine currently assigned positions/dimensions of the respective UI elements, as specified in a CSS file describing a UI presentation of the webpage. Further, during the filtering phase of the detection of UI layout issues for the webpage, the functions of generating screenshots (reference numeral 232; see
It was further described herein that a container element (e.g., the box 208; see
For example, during the filtering phase of the detection of UI layout issues for the webpage, the functions of generating screenshots (reference numeral 232; see
Although features are shown and described herein with reference to particular illustrative embodiments of the disclosed techniques, such features may be included and are hereby included in any of the illustrative embodiments and their variants. Thus, it is understood that features disclosed in connection with any illustrative embodiment are included as variants of any other illustrative embodiment. Further, the disclosed techniques or portions thereof may be embodied as a computer program product including one or more non-transient, computer-readable storage media, such as a magnetic disk, magnetic tape, compact disk, digital versatile disk (DVD), optical disk, flash drive, solid state drive (SSD), secure digital (SD) chip or device, application specific integrated circuit (ASIC), field programmable gate array (FPGA), and/or the like. Any number of computer-readable media may be used. The media may be encoded with instructions that perform the process or processes described herein when executed on one or more computers, processors, or processing circuitry. Such media may be considered articles of manufacture or machines, and may be transportable from one machine to another.
As used throughout this document, the words “comprising,” “including,” “containing,” and “having” are intended to set forth certain items, steps, acts, elements, or aspects of something in an open-ended fashion. Further, as employed herein, the word “set” or “subset” means one or more of something. This is the case regardless of whether the phrase “set of or” subset of is followed by a singular or plural object, and regardless of whether it is conjugated with a singular or plural verb. In addition, although ordinal expressions such as “first,” “second,” “third,” and so on, may be used as adjectives herein, such ordinal expressions are used for identification purposes, and, unless specifically indicated, are not intended to imply any ordering or sequence. Thus, for example, a “second” event may take place before or after a “first event,” or even if no first event ever occurs. In addition, an identification herein of a particular element, feature, or act as being a “first” such element, feature, or act should not be construed as requiring that there must also be a “second” or other such element, feature, or act. Rather, the “first” item may be the only one. Although certain illustrative embodiments are disclosed herein, it is understood that these embodiments are provided by way of example, and that the disclosed techniques are not limited to these particular embodiments.
While various embodiments of the disclosure have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure, as defined by the appended claims.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN2018/120070 | 12/10/2018 | WO | 00 |