Patent Application
20140280880
The present disclosure relates generally to cloud computing and mobile testing; more particularly, to automated systems and methods for capturing and analyzing real-time information of actual user experience on websites and using web-based applications.
Various platforms for load testing of websites and web-based applications are commercially available. For example, U.S. Pat. No. 7,844,026 describes a real-time analytics tool that allows businesses to gather and display live performance data obtained from running a complex test composition on a target website or web application. This tool performs a load test utilizing large numbers (e.g., hundreds of thousands) of virtual users, providing business intelligence data results that permit a business to pinpoint performance bottlenecks and potential areas of stress in a web application.
Organizations are also interested in real user measurement (RUM) data analysis that captures and collects data about present, real user experiences when users visit and use a website or web application. For example, businesses engaged in e-commerce are often interested in a performance metric known as the “bounce rate”, which is a ratio of the number of visitors to a website who immediately leave the site after viewing only one page, versus users who click on an actionable item or icon (e.g., to place an item in a shopping cart). Since there is a strong correlation between the speed of a website (e.g., the time to load a webpage) and the probability of a user bouncing, real-time analytics that gives businesses and developers insight into RUM across all browsers and locations is very valuable.
Online analytical processing (OLAP) of collected data has recently given rise to the use of analytic dashboards as a way to visualize key performance indicators of a website or web application. Dashboards usually consist of a series of graphics, charts, gauges and other visual indicators that can be monitored and interpreted. Analytical dashboards typically support interactions with the data, such as drilling down into the underlying details. One visual indicator typically found in dashboards is a histogram. A histogram is a type of graph widely used in statistics to visually interpret numerical data by indicating the number of data points that lie within a range of values, commonly referred to as a class or bin. The frequency of the data that falls in each class is depicted by the use of a bar. The height of the bar corresponds to the relative frequency of the amount of data in the class. In other words, the higher the bar, the greater the frequency of the data. Conversely, the lower the bar the lower the frequency of the data. The bars in a histogram are arranged and displayed in the order that the classes occur.
One of the problems with providing visual indicators such as histograms in real-time analytic dashboards is that statistical information, such as a percentile calculation, needs to be performed in real-time, concurrently with the on-going collection of data, which can involve tens or hundreds of millions of real user measurements. For example, a typical way to compute a percentile is to first sort all of the data points in ascending order, i.e., smallest data points to the largest. The nth percentile is then determined by the corresponding location in the order. By way of example, if 100 data points are sorted in ascending order, then the tenth percentile is the tenth data point in the order. But with extremely large data sets the computing power and memory requirements needed to store and sort all of the data can quickly exceed reasonable bounds.
The present disclosure will be understood more fully from the detailed description that follows and from the accompanying drawings, which however, should not be taken to limit the invention to the specific embodiments shown, but are for explanation and understanding only.
In the following description specific details are set forth, such as data types, metrics, devices, functions, etc., in order to provide a thorough understanding of the subject matter disclosed herein. However, persons having ordinary skill in the relevant arts will appreciate that these specific details may not be needed to practice the present invention. It should also be understood that the elements in the figures are representational, and are not necessarily drawn to scale in the interest of clarity.
References throughout this description to “one embodiment”, “an embodiment”, “one example” or “an example” means that a particular feature, structure or characteristic described in connection with the embodiment or example is included in at least one embodiment. The phrases “in one embodiment”, “in an embodiment”, “one example” or “an example” in various places throughout this description are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures or characteristics may be combined in any suitable combinations and/or sub-combinations in one or more embodiments or examples.
In the context of the present application, the term “cloud” broadly refers to a collection of machine instances, storage and/or network devices that work together in concert. The term “cloud computing” refers to a paradigm in which machine, storage, and application resources exist on a “cloud” of servers. In cloud computing shared resources, software and information are provided on-demand, like a public utility, via the Internet. Thus, cloud computing provides computation, data access, and storage resources without requiring users to know the location and other physical details of the computing infrastructure. Cloud computing is closely related to grid computing, which refers to the concept of interconnecting networked computers such that processing power, memory and data storage are all community resources that authorized users can utilize for specific tasks.
The term “server” broadly refers to any combination of hardware or software embodied in a computer (i.e., a machine instance) designed to provide services to client devices or processes. A server therefore can refer to a computer that runs a server operating system from computer-executable code stored in an associated memory, and which is provided to the user as a virtualized or non-virtualized machine; it can also refer to any software or dedicated hardware capable of providing computing services. In the context of the present disclosure, “Result” servers (also referred to as “collector” servers) are servers deployed and utilized to receive real-user measurement data sent from a user's client device. Each of the collectors process and aggregate the data items received.
The term “real-time” refers to a level of computer responsiveness that a user senses as sufficiently immediate or that enables the computer to keep up with some external process (for example, to present visualizations of load test results as it constantly changes). Thus, real-time is a mode of computer operation in which the computer collects data, analyzes or computes with the data, reports (e.g., visually displays) and/or stores the results nearly instantaneously, i.e., within seconds or milliseconds.
In the context of the present disclosure, the term “beacon” refers to data related to a user” experience on a particular website or web application, collected by a library (e.g., a JavaScript library) running on the browser of a client device, and sent via Hypertext Transfer (or Transport) Protocol (HTTP) to a server. The server receiving the beacon information may aggregate that data along with similar data received from other users accessing the same website or web application. Any HTTP headers sent by the browser as part of the HTTP protocol are also considered part of the beacon. In a sense, a beacon may be thought of as a page view on a website, but without a corresponding page. For every user who visits that website, the browser running the library on the user's client device measures various metrics and records data that is then “beaconed” back to a results server in real-time as the user navigates through or uses the website.
A “data bucket” or “bucket” refers to a type of data buffer, data container, block of memory, or file on a disk that contains data. In the present disclosure, data buckets are arranged in a set or array, with each data bucket containing a count of a number of data values falling within a predetermined range. A given data bucket may be empty, or non-empty. The set or array of data buckets are typically arranged in an ascending order such that all of the data buckets span a full range of data values expected to be received for a particular data set or data type, e.g., from the lowest data value to the highest data value. Each of the data buckets are defined with predetermined value ranges such that a received data value will fall within a single one of the data buckets in the set or array.
In one embodiment, a system and method for fast percentile approximation using data bucketing is described. A results server (i.e., collector) aggregates beacon data received from a plurality of client devices associated with users visiting or accessing a particular website or web application. In one embodiment, the width or range of each data bucket used for collecting data received and aggregated from the user beacons is set to be equal, bucket-to-bucket. In another embodiment, each of the data buckets is assigned a predetermined variable-width or range. Meaningful histograms suitable for display in a real-time analytic dashboard may be generated by appropriate scaling of the variable-width data buckets in accordance with a scaling algorithm.
The information collected and periodically sent to server 20 may include such metrics as web page load time, total load time, number of web pages accessed, average load time per page, etc. The specific metrics and data collected and sent to server may vary depending on the information of interest to the business or enterprise owning the website. In addition, the periodicity or interval for sending the data collected may vary case-to-case. In one embodiment, metrics such as page load times and average load time may be sent for each page accessed by the user. In other embodiments, metrics and data collected may be beaconed to server 20 on a predetermined time interval, e.g., every 100 ms.
In one embodiment clouds 11 and 16 may comprise the same public network (i.e., the Internet). Alternatively, clouds 11 and 16 may comprise a variety of different public and/or private networks.
It is appreciated that server 20 may receive beacons containing metrics and other performance data from a multitude of different client devices, each of which may be located in a different geographic area. In other cases, results server 20 may receive metrics and data from a multitude of different client devices located in the same geographic region (e.g., San Francisco or Boston). It is appreciated that a hierarchy of servers may be arranged to collect and consolidate data and metrics received from millions, or even billions, of client devices accessing the same website or web application at the same time. All of this data is sent to a ResultService reader/writer (R/W) unit 17 that aggregates the total data received and stores it in a database 18, making it accessible to a main computer instance 19, which implements a real-time analytic dashboard for visual presentation of the RUM results stored in database 18. It is appreciated that in other embodiments the aggregating unit may comprise another server, or other computing device.
In the example shown, main instance 19 is a virtual machine deployed on a server provided that communicates with a browser application. In one embodiment, main instance 19 may include a results server or service which reads data from database 18 and serves it to a web application, which in turn formats the data and serves it to an analytic dashboard in the browser. In operation, main instance 19 executes the coded sequence of computer executed steps (e.g., from code stored in a memory) that collects and aggregates all of the user beacon data and metrics for a particular website or web application. The computer program executed in the main instance 19 may also allocate the results server resources required for the RUM across one or more different cloud providers. The same application that allocates/verifies server resources may also verify that the allocated servers are operational to collect and aggregate the RUM metrics and data. The main instance may also execute code that implements the RUM results aggregation steps and storage of results data in database 18. In addition, main instance 19 may implement the analytic dashboard utilized for visualized display of the aggregated results.
Although
Persons of skill in the art will understand that the software which implements the RUM results analysis platform may also be downloaded to the user's laptop computer 13 or implemented on a separate hardware appliance unit located either at the user's premises or anywhere in clouds 11, 16, or another cloud.
In the example of
A percentile approximation for a given data set is computed by applying the formula shown at the bottom of
Persons of skill in the art will appreciate that the use of data bucketing for quickly computing percentile approximation for a large data set, as described above, can take place in real-time across thousands of computing devices, each in a different location, as the real user measurements are being collected and aggregated by the results servers in the cloud. It is further appreciated that finer granularity in data analysis calculations may be achieved by increasing the number of data buckets fixed in the array. Moreover, since the number of data buckets is finite and data counts from corresponding buckets are easily and quickly accumulated, the size of the data set is not a constraint. Thus, data analysis calculations such as percentile approximation, median load time, etc., can be performed in real-time.
It is appreciated that the hierarchal arrangement of servers shown in
For percentile calculations, the bucket containing the desired percentile is identified. (Block 43) By way of example, for a 50th percentile calculation this may be accomplished by dividing the total number of data points received at any given time by two, and then adding the counts together in ascending order (i.e., from the lowest or first data bucket on up) until the data bucket is identified as containing the 50th percentile data point. Finally, the position in the identified bucket of the data point at the desired percentile is approximated. (Block 44) In one embodiment, the approximation assumes a linear distribution of data points within the range defined for the bucket identified as containing the desired percentile data point. By way of illustration, if we assume that a fifth data bucket has 10,000 data points, a range of 1 second, and is identified as containing the 50th percentile data point for an array of data buckets containing a total of 1 million data points. And if we assume that the total sum of the first four data buckets is 495,000, then the 50th percentile data point may be approximated as being located midway (i.e., ½ second from the bottom value) in the fifth data bucket.
In another embodiment, the width or range of each data bucket in a set or array of buckets is not uniform. That is, the range of each data bucket may vary across the set. For instance, it may be desirable for certain data sets to have finer granularity (smaller range) at the lower end of the data type being measured and lower granularity (larger range) at the upper end. Consider the example of web page load times measured from real users. Such measurements may typically have data points mostly clustered around the lower end (e.g., 0-4 s) and fewer data points at the higher end (e.g., 10-20 s). Therefore, the data bucket widths may be defined to be very small at the lower end (e.g., 100 ms) and much larger at the higher end (e.g., 2-5 s). In other words, the width of the data buckets in a set or array may be tailored to the specific properties of the type of data being measured. In the case of web page load times, enterprises are understandably concerned about fast load times; therefore, they may define the buckets at the lower end to have finer granularity. It should be kept in mind, however, that the total number of buckets still remains fixed even though individual buckets may have different widths or ranges.
The initial step (block 51) in the example process shown in
Continuing with this example, the next data bucket 65 is 300 ms wide but contains no data points. Therefore, the next 300 ms interval on the histogram of
Buckets 67, 68 and 69 all have widths that are greater than the minimum bar width. As shown, the histogram bars representing buckets 67-69 have widths that are equal to the width of their corresponding bucket. Thus, the last three (right-most) bars illustrated in
It should be understood that elements of the disclosed subject matter may also be provided as a computer program product which may include a machine-readable medium having stored thereon instructions which may be used to program a computer (e.g., a processor or other electronic device) to perform a sequence of operations. Alternatively, the operations may be performed by a combination of hardware, firmware, and software. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks. ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, or other type of machine-readable medium suitable for storing electronic instructions.
Additionally, although the present invention has been described in conjunction with specific embodiments, numerous modifications and alterations are well within the scope of the present invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.
