Patent Application
20150370439
Embodiments of the subject matter described herein relate generally to computer systems, and more particularly relate to systems and methods for managing the scrolling of web pages and other such content in such computer systems.
Recent years have seen significant advances in user interface design in the context of computing devices, particularly those devices—such as tablets and smart phones—that are often used for browsing web content available over the Internet. There are still many opportunities for improvement, however.
For example, one of the core requirements of any web browser is the ability to scroll through web pages that are too large to be displayed in their entirety on the device screen. Ideally, scrolling should proceed smoothly and at an appropriate speed such that the user does not experience any hesitation or other unusual animation artifacts. This can be a problem, particularly in resource limited devices, because the relatively small memory of such devices prevents a sufficient number of the displayable components to be stored and manipulated in real-time.
Furthermore, the manner in which displayable content is rendered and scrolled in a browser typically requires many redraw events, which can be taxing for even powerful processors and graphics processing units (GPUs). In addition, the browser will often not know, until the scroll animation has completed, which component is to be displayed at the end of the scroll event. As a result, such a browser will often display one or more dummy “placeholder” graphics during scrolling.
While techniques exist for providing limited scrolling using a scrolling region programmatically provided within the web page, e.g., via cascading style sheet (CSS) transitions, such methods can be slow, and are not extensible—i.e., they do not allow for rich customization of animation behavior.
Accordingly, systems and methods are desired for improving the scrolling of content in the context of web pages and the like.
A scrolling method in accordance with one embodiment includes producing a render tree associated with a plurality of web resources and a plurality of displayable components; providing a subset of the plurality of displayable components to a graphics processing unit such that each of the displayable components has its own corresponding layer; receiving a scroll gesture indicative of a request to scroll the plurality of displayable components; determining a scroll behavior based on the scroll gesture; and sequentially modifying and rendering the subset of the plurality of displayable components based on the scroll behavior.
A computing system in accordance with one embodiment includes a processing system and a memory, wherein the memory comprises computer-executable instructions that, when executed by the processing system, cause the computing system to: produce a render tree associated with a plurality of web resources and a plurality of displayable components; provide a subset of the plurality of displayable components to a graphic processing unit such that each of the displayable components has its own corresponding layer; receive a scroll gesture indicative of a request to scroll the plurality of displayable components; determine a scroll behavior based on the scroll gesture; and sequentially modify and render the subset of the plurality of displayable components based on the scroll behavior.
A more complete understanding of the subject matter may be derived by referring to the detailed description and claims when considered in conjunction with the following figures, wherein like reference numbers refer to similar elements throughout the figures.
Embodiments of the subject matter described herein generally relate to an extensible platform that provides GPU-optimized scrolling in the context of a web browser or the like. By combining mathematically specified scrolling behavior with the “recycling” of the render tree at the GPU, a smooth scrolling experience can be provided, even when a very large number of items are displayed.
Referring now to the conceptual block diagram of
Browser 230 may correspond to any of the variety of web browsers known in the art, such as Microsoft Internet Explorer, Mozilla Firefox, Google Chrome, Apple Safari, Opera, or the like. Similarly, browser 230 may implement a variety of application programming interfaces (APIs) and include any suitable layout engine. In that regard, while the various examples presented herein are often described in the context of the Webkit layout engine and its commands, the invention is not so limited. Other layout engines, such as Trident and Gecko, may be employed. Browser 230 will generally include a user interface, a browser engine, a rendering engine, a Javascript interpreter, and a variety of other functional components known in the art.
As illustrated, content is displayed by browser 230 within a rectangular region defined by edges 201, 202, 203, and 204. Because the area defined by edges 201-204 is significantly smaller than the total area of content 100, only a portion of content 100 can be displayed by computing device 200 at any particular time. In
Thus, for example, movement of input object 202 upward (relative to
After input object 202 is removed from contact region 204 (in many cases with a “flicking” action), the displayed content 100 will generally translate in accordance with a particular scrolling behavior and then eventually “settle” to a stationary position. Certain “bounces” or other animations (e.g., “pull to refresh”) may also be implemented. As will be understood, computing device 200 will generally be capable of interpreting a wide range of gestures, including pinch movements and the rotation and translation of multiple input objects 202 (e.g., “multi-touch”). In general, then, computing device 200 is capable of determining (1) whether input object 202 is “in contact” with display 206, and (2) the effective position (or positions) of any contact regions 204.
As will be described in further detail below, scrolling of content 100 within browser 230 is accomplished not through the browser's built-in browser functionality, but by a “scroller” object (or objects) incorporated (via HTML/CSS/Javascript) into the displayed region of browser 230. The scroller manages the movement of the content 100 via efficient use of a graphics processing unit (GPU) provided within computing device 200.
More particularly, referring to
In general, browser 404, in conjunction with scroller 406, any plug-ins 407-409, and GPU 310, are configured to parse web resources 402 to first construct a document object model (DOM) tree. That is, HTML tags (“<body>”, “<table>”, etc.) within resources 402 are used to create a “content tree” corresponding to the displayable content. The corresponding CSS file is then parsed and combined with the content tree to produce a “render tree.” The render tree includes shape information for the various displayable content as well as visual attributes, such as colors and dimensions. Such a render tree 700 is illustrated, conceptually, in
After construction of the render tree, a “layout” process is performed to determine where each node of the render tree 700 should appear on the display. All or a portion of the render tree is provided to GPU 310, which then “rasterizes” (pixel by pixel) the render tree 700 to produce the final image for the display.
As noted above, one problem with known scrolling schemes is that, when the render tree 700 contains a large number of displayable components, the memory of the computing device (306 in
Forcing the displayable components to reside on individual layers may be accomplished in a number of ways, depending upon the nature of GPU 310 and browser 404. In one embodiment, in which browser 404 utilizes the Webkit library, each displayable component is subjected to a webkit matrix3d transformation that specifies the relative position of each displayable component. That is, the 4×4 matrix passed to the matrix3d function (which controls translation, scaling, and rotation) is selected to effect a desired motion during the scroll event. In accordance with one embodiment, an HTML/Javascript “surface.dom.style” transform is used, wherein the 4×4 matrix includes “offsetX” and “offsetY” variables that specify the position of a displayable component during scrolling, as in the following command: surface.dom.style[STYLES.transform]=‘matrix3d(1,0,0,0,0,1,0,0,0,0,1,0,’+offsetX+‘,’+offsetY+‘, 0, 1)’. In this example, the system forces a layer to be promoted (at a GPU level) by using translate3d( )or matrix3d( ) command. In other embodiments, a more general command, such as the “will-change” CSS selector may be used (see, e.g., the W3C (standardization committee) document http://www.w3.org/TR/2014/WD-css-will-change-1-201404294/).
Referring again to
Once the system knows the precise location of each of the displayable components during the scroll event, the system (via GPU 310) renders and composites the displayable components in accordance with the known scroll behavior while “recycling” the render tree. That is, referring again to
The foregoing description is merely illustrative in nature and is not intended to limit the embodiments of the subject matter or the application and uses of such embodiments. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the technical field, background, or the detailed description. As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations, and the exemplary embodiments described herein are not intended to limit the scope or applicability of the subject matter in any way.
For the sake of brevity, conventional techniques related to touchscreen systems, HTML and related technologies (Javascript, CSS, the DOM, etc.), GPUs, and other functional aspects of the systems (and the individual operating components of the systems) may not be described in detail herein. In addition, those skilled in the art will appreciate that embodiments may be practiced in conjunction with any number of system and/or network architectures, data transmission protocols, and device configurations, and that the system described herein is merely one suitable example. Furthermore, certain terminology may be used herein for the purpose of reference only, and thus is not intended to be limiting. For example, the terms “first”, “second” and other such numerical terms do not imply a sequence or order unless clearly indicated by the context.
Embodiments of the subject matter may be described herein in terms of functional and/or logical block components, and with reference to symbolic representations of operations, processing tasks, and functions that may be performed by various computing components or devices. Such operations, tasks, and functions are sometimes referred to as being computer-executed, computerized, software-implemented, or computer-implemented. In practice, one or more processing systems or devices can carry out the described operations, tasks, and functions by manipulating electrical signals representing data bits at accessible memory locations, as well as other processing of signals. The memory locations where data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties corresponding to the data bits. It should be appreciated that the various block components shown in the figures may be realized by any number of hardware, software, and/or firmware components configured to perform the specified functions. For example, an embodiment of a system or a component may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. When implemented in software or firmware, various elements of the systems described herein are essentially the code segments or instructions that perform the various tasks. The program or code segments can be stored in a processor-readable medium or transmitted by a computer data signal embodied in a carrier wave over a transmission medium or communication path. The “processor-readable medium” or “machine-readable medium” may include any non-transitory medium that can store or transfer information. Examples of the processor-readable medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an erasable ROM (EROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, or the like. The computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic paths, or RF links. The code segments may be downloaded via computer networks such as the Internet, an intranet, a LAN, or the like. In this regard, the subject matter described herein can be implemented in the context of any computer-implemented system and/or in connection with two or more separate and distinct computer-implemented systems that cooperate and communicate with one another.
While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or embodiments described herein are not intended to limit the scope, applicability, or configuration of the claimed subject matter in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing the described embodiment or embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope defined by the claims, which includes known equivalents and foreseeable equivalents at the time of filing this patent application. Accordingly, details of the exemplary embodiments or other limitations described above should not be read into the claims absent a clear intention to the contrary.
This application claims priority to U.S. Provisional Patent Application No. 62/016,441, filed Jun. 24, 2014, the contents of which are hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62016441 | Jun 2014 | US |
