Patent Application
20250138992
The present disclosure relates to error evaluation, and, in particular, to a system and method for error reproduction in web-based applications.
The problem of isolating and analyzing user sessions and paths on websites is significant across numerous industries, including e-commerce, user experience design, and web development. It impacts website owners, marketers, and developers who seek to understand how users navigate and encounter bugs/errors on their websites that generate losses in sales and revenue. Correctly identifying user experience degradations, assessing website structure effectiveness and design responsiveness within the vast amounts of session data collected is a challenge.
This background information is provided to reveal information believed by the applicant to be of possible relevance. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art or forms part of the general common knowledge in the relevant art.
The following presents a simplified summary of the general inventive concept(s) described herein to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to restrict key or critical elements of embodiments of the disclosure or to delineate their scope beyond that which is explicitly or implicitly described by the following description and claims.
A need exists for understanding errors and bugs on websites at the most granular reproducible level, to enhance website performance, and find the right guidance for all issues.
In accordance with a first aspect, there is provided a computer-implemented method for identifying a sequence of steps that reproduces a reported application error, comprising the steps of: categorizing, by a processor, error information associated with the reported application error experienced by a plurality of users in a plurality of user sessions, the error information comprising, for each user session, one or more user events and associated application information, to determine a plurality of associated grouped events; and isolating, by the processor, from the plurality of associated grouped events, one or more designated grouped events causing the reported application error; and identifying, by the processor, from the one or more designated grouped events, a direct path causing the reported application error.
In some embodiments, the one or more user events comprise at least one of: a click, a keyboard interaction, or a scroll.
In some embodiments, the method further comprises the steps of, before said categorizing: capturing user activity from said plurality of user sessions; and converting said user activity into said one or more user events and associated application information.
In some embodiments, the method further comprises the step of: displaying, by the processor, the sequence of steps corresponding to the direct path causing the reported application error.
In some embodiments, categorizing comprises the steps of: pre-processing, by the processor, the one or more user events and associated application information into a plurality of embeddings; and clustering, by the processor, the plurality of embeddings into the plurality of associated grouped events by performing semantics clustering analysis via one or more machine learning models.
In some embodiments, the one or more machine learning models uses a transformer architecture.
In some embodiments, the associated application information comprises at least one of: cascading style sheet (CSS) elements, hypertext markup language (HTML) elements, a uniform resource locator (URL), text, and screen recordings.
In some embodiments, the pre-processing is done by, at least in part, combining a class and identification (ID) of an HTML element to build one or more CSS groupings.
In some embodiments, the pre-processing comprises utilizing an image source of a user-event.
In some embodiments, the pre-processing comprises masking numerical portions of the text and CSS elements.
In accordance with another aspect, there is provided a system for identifying a sequence of steps that reproduce a reported application error, comprising: a processor for executing instructions; a non-transitory memory coupled to the processor storing instructions that when executed by the processor configures the system to perform the method of: categorizing, by a processor, error information associated with the reported application error experienced by a plurality of users in a plurality of user sessions, the error information comprising, for each user session, one or more user events and associated application information, to determine a plurality of associated grouped events; and isolating, by the processor, from the plurality of associated grouped events, one or more designated grouped events causing the reported application error; and identifying, by the processor, from the one or more designated grouped events, a direct path causing the reported application error.
In accordance with another aspect, there is provided a non-transitory computer-readable medium having statements and instructions stored thereon to be executed by a digital processor to automatically: categorize error information associated with the reported application error experienced by a plurality of users in a plurality of user sessions, the error information comprising, for each user session, one or more user events and associated application information, to determine a plurality of associated grouped events; and isolate from the plurality of associated grouped events, one or more designated grouped events causing the reported application error; and identify, from the one or more designated grouped events, a direct path causing the reported application error.
In some embodiments, the one or more user events comprise at least one of: a click, a keyboard interaction, or a scroll.
In some embodiments, the statements and instructions cause the digital processor to automatically, before said categorizing: capture user activity from said plurality of user sessions; and convert said user activity into said one or more user events and associated application information.
In some embodiments, the statements and instructions further cause the digital processor to automatically: display the sequence of steps corresponding to the direct path causing the reported application error.
In some embodiments, categorizing comprises the steps of: pre-processing the one or more user events and associated application information into a plurality of embeddings; and cluster the plurality of embeddings into the plurality of associated grouped events by performing semantics clustering analysis via one or more machine learning models.
In some embodiments, the one or more machine learning models uses a transformer architecture.
In some embodiments, the associated application information comprises at least one of: cascading style sheet (CSS) elements, hypertext markup language (HTML) elements, a uniform resource locator (URL), text, and screen recordings.
In some embodiments, the pre-processing is done by, at least in part, combining a class and identification (ID) of an HTML element to build one or more CSS groupings.
In some embodiments, the pre-processing comprises utilizing an image source of a user-event.
Other aspects, features and/or advantages will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the accompanying drawings.
Several embodiments of the present disclosure will be provided, by way of examples only, with reference to the appended drawings, wherein:
Elements in the several drawings are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be emphasized relative to other elements for facilitating understanding of the various presently disclosed embodiments. Also, common, but well-understood elements that are useful or necessary in commercially feasible embodiments are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure.
Various implementations and aspects of the specification will be described with reference to details discussed below. The following description and drawings are illustrative of the specification and are not to be construed as limiting the specification. Numerous specific details are described to provide a thorough understanding of various implementations of the present specification. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of implementations of the present specification.
Furthermore, numerous specific details are set forth in order to provide a thorough understanding of the implementations described herein. However, it will be understood by those skilled in the relevant arts that the implementations described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the implementations described herein.
In this specification, elements may be described as “configured to” perform one or more functions or “configured for” such functions. In general, an element that is configured to perform or configured for performing a function is enabled to perform the function, or is suitable for performing the function, or is adapted to perform the function, or is operable to perform the function, or is otherwise capable of performing the function.
When introducing elements of aspects of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. The term “exemplary” is intended to mean “an example of.” The phrase “one or more of the following: A, B, and C” means “at least one of A and/or at least one of B and/or at least one of C.”
Errors may affect the actions of users in many ways. For instance, as a result of an error, the user may simply cease interacting with the computer system. The user may also refuse or be unable to use the computer system as they intended (e.g., they may be prevented from making a purchase). They may also make a complaint to administrative users. (These should, of course, be understood as merely some examples of possible effects, and not as an exhaustive list.) As would be evident, some effects may be more significant/severe than others. Also, as would be clear, a single error may have more than one effect. For instance, if a user is unable to make a purchase, they may quit (i.e., cease interacting with the computer system). Further, as discussed above, some errors may have no discernable effect on a user's actions and may thus be relatively non-severe. It should be noted, however, that different users may respond differently to different errors. Thus, it may be preferable to gather data related to many users' actions in response to a certain error. More importantly, it can be difficult to accurately isolate and identify the direct path taken that causes the issue outright on the website.
The present disclosure provides a system and method for error reproduction in a computer system, including network based applications, such as e-commerce applications. For example, the system and method allow to effectively and accurately analyze user sessions and paths associated with website interactions that lead to lost revenue from application errors such as website bugs. The system and method provide improved means of correctly identifying a sequence of steps or a session path that reproduces an error in a given software environment. This includes capturing and analyzing a plurality of user events (e.g., clicks, keyboard events, scrolls, etc.) in a plurality of user sessions with associated application data (e.g., Cascading Style Sheets (CSS) elements, HTML elements, etc.). In some embodiments, the system and method use the error information in an advanced and efficient way to understand and optimize user interactions on websites. While the following disclosure is mainly directed to e-commerce applications, the skilled person in the art will appreciate that other types of applications may also benefit from the disclosure.
Note that the term ‘error’ as used herein should be interpreted broadly. ‘Errors’ includes, but are not limited to, ‘bugs’, ‘glitches’, flaws in code, problems resulting from unexpected user input, logic errors, display errors, interface errors, status and/or network communication errors, and/or anything else that results in less-than-optimal performance of the computer system. Additionally, the term ‘error’ may include errors that prevent users from accessing the computer system at all. Similarly, the term ‘computer system’ should be interpreted broadly, as including without limitation websites, games, computer programs, downloaded software, on-line or cloud-based software, distributed systems, and hardware on which the software may be run. In some embodiments, the system of the present disclosure may not distinguish between hardware and software errors, while in other embodiments, such distinctions may be made.
In some embodiments, the user-devices 104a-104n typically take the form of any computing or digital processing device, including for example, but not limited to, a desktop computer, a laptop computer, a tablet, a smartphone, a smartwatch or the like. The user devices 104a, . . . 104n are each communicatively coupled, directly or indirectly, to a server 106 via a network 108. Although a single server 106 is illustrated in
In some embodiments, the memories 208 and 216 may comprise any known form of computer memory or storage. This may include, for example, but without limitation, storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory, solid state memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Any method, application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.
In some embodiments, the CPU(s) 206 and 214 may include any known type of digital processor or the like. This can include general purpose processors (e.g., x86 or the like), embedded processors, or programmable processors (e.g., field programmable gate array processors or FPGA). Further, unless the context clearly indicates otherwise, any processor set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified.
The network component 212 and 218 each may be configured to enable the computing device to communicate and/or exchange data with other components, to access and connect to network resources, to serve applications, and/or perform other computing applications by connecting to a network 220. The network 220 can be a single network or multiple connected networks capable of carrying data, including the Internet, an Ethernet network, integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, Bluetooth, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, to name a few examples.
The input/output components (e.g., I/O components 210) of the user device 204 allow communication with a human user via an associated input, output, or input/output device such as a keyboard, a mouse, a touchscreen, and display. The display may be used to generate a graphical user interface (GUI) or the like. In the case where the user device 204 is used by an administrator, the GUI is used to provide information about the application errors encountered and their analysis.
As mentioned above, in some embodiments, the system may be configured using a client-server model or the like. The user device 204 may comprise a client-side application 222 in memory 208 which, upon being executed by the CPU(s) 206 allow the user device 204 to communicate with the server 202 via the network component 212. Other embodiments may have the user device 204 connecting to server 202 via a web browser 224 or the like. The server 202 may comprise in the memory 216 a server-side application 226 that allows the CPU(s) 214 to respond to or communicate with the user device 204 via the network component 218. In addition, the error analysis module 232 may use the captured session information 234 retrieved, in some embodiments, from the user device 204 to analyze and determine the steps needed to reproduce the application error.
In some embodiments, user session information may be captured from recorded user sessions (e.g., screen recordings), for example via a data collector module 230 stored in memory 208 of the user device 204. The data collector module 230 may be a separate application or script accessed or downloaded via a web page.
In some embodiments, different modules/instructions may be combined or distributed on different devices, in accordance with different implementations.
In some embodiments, the server 202 and/or user device 204 may further be configured to communicate with one or more 3rd-party databases 228 via network 220. This may include, for example, one or more 3rd-party servers or databases relating to different services, for example email servers, scheduling systems, or the like.
In some embodiments, the user-event information 302 and associated application information 304 may be captured from recorded user sessions. For example, in some embodiments, the user-event information may be captured on the client side. For example, the session information may be collected from users accessing the e-commerce application. A data collector module 230 may be provided as a script that is downloaded and run as part of the e-commerce application and/or embedded in a webpage of the e-commerce application. The data collector script may be an initial script downloaded and run in order to capture any potential errors resulting when loading the e-commerce application. The browser information may provide information about the browser version, plug-ins, etc. as well as information about the operating system of the user's device. The data collection method may capture initial document object model (DOM) of the e-commerce application. The DOM provides an initial starting point view of the application. User events or user-event information, such as mouse movement, clicks, keyboard inputs, buttons clicked, scrolling, etc. are captured, which in coordination with the initial DOM and browser information may be used to re-create the user's particular session. Application information such as HTML requests and the results, possibly including resulting errors, are captured as well. The captured information is transmitted to the server (e.g., server 202) for further processing. In some embodiments, the recorded user session information may be used to produce a human-viewable reconstruction of the user session. The user reconstruction may be in a video or image format.
In some embodiments, an image source of a “click” event (for example from a screen recording) may be used and analyzed instead of using the non-visual “click” event information. Typically, one or more pre-processing techniques may be used to transform and normalize the collected raw user-event information and associated application information into a plurality of vector embeddings processable by the one or more machine learning models. These are stored in a plurality of documents or files 504. The pre-processed embeddings are then used to train the one or more machine learning models 406.
Once pre-processed, the embeddings are sent to one or more machine learning models 406 trained to perform text classification by learning the patterns existing in all of the user-event information and associated application information. In some embodiments, the machine learning models 406 are semantics clustering models trained to classify or group the embeddings into different topics. This is accomplished by using the semantic meaning of words to quantify similarity between different pieces of text. In some embodiments, the machine learning models 406 may comprise one or more neural networks, including using a transformer deep learning architecture configured to perform natural language processing. The machine learning models 406 are trained/configured to provide one or more associated grouped events across the different user sessions. The grouped events are categorized into easily identifiable categories.
The analytics engine 408 may use the grouped events generated by the ML models 406 to identify and isolate (508) the session paths (e.g., series of steps 506a, 506b, . . . 506N) preceding the identified application errors 1, 2, . . . . N. The series of steps may be provided to an administrative user in the form of a graphical user interface.
While the present disclosure describes various embodiments for illustrative purposes, such description is not intended to be limited to such embodiments. On the contrary, the applicant's teachings described and illustrated herein encompass various alternatives, modifications, and equivalents, without departing from the embodiments, the general scope of which is defined in the appended claims. Information as herein shown and described in detail is fully capable of attaining the above-described object of the present disclosure, the presently preferred embodiment of the present disclosure, and is, thus, representative of the subject matter which is broadly contemplated by the present disclosure.
