Computing devices display information via user interfaces. User interfaces often include a layered collection of elements or “views” to be drawn to a screen. It is common for user interface systems to render views composed of overlapping rectangles or boxes that contain and/or arrange content such as images or text. Each view may have properties, such as a background color or image. Such views or elements may be organized in a hierarchical logical data structure in memory, such as in a tree. For example, web browsers conventionally use a Document Object Model (“DOM”) to organize and represent objects in web pages (e.g., HTML, XHTML, and XML documents), and native display systems such as iOS® and Android™ can use similar hierarchical models to represent views for display on various computing device user interfaces. A rendering engine executes commands to draw the rectangles or boxes (or, other polygons, e.g., triangles) with specified coordinates to the screen according to the specified views.
The amount of time available to render a frame containing a full screen of views or elements in a user interface can be limited, especially in the context of redrawing the screen (e.g., in response to user input such as scrolling). The amount of time spent rendering each frame is typically proportional to the number of rendering commands sent to a display component, e.g., a rendering engine, graphics processor, etc. Rendering commands are instructions to render (e.g., draw) a particular view. If too many commands are sent, the display system may not be able to keep up; the result can be dropped frames that can cause a visual stuttering effect or choppiness instead of smooth, responsive movement, e.g., when a user scrolls the resulting display. The disclosure is directed to reducing the number of rendering commands to be sent to render a layered collection of elements to be drawn to a screen, such as a mobile device screen. It is common for one element or a series of elements in the foreground to overdraw and thus occlude an element in the background. As a result, some elements are not visible, even if they are drawn to the screen. The inventors of the present disclosure have determined that properties of occluded elements or views that would cause the device to draw a background image or color not visible to an end user can be changed (e.g., un-set or resized) or removed, thereby reducing the number of rendering commands to be sent and thus improving rendering performance.
The present disclosure describes automatic overdraw reduction before rendering, including systems and methods for rendering pages more quickly and with less overhead by automatically avoiding rendering an element on a page that is fully or partially occluded. Some implementations detect when an element is fully occluded (e.g., a background image or color that is covered by one or a series of foreground items) and remove the element or a property of the element so that it is not rendered. In some implementations, if an element would not be rendered on a screen, a property of the element (e.g., a background) is modified such as by un-setting the property. In some implementations, if the element is partly occluded as a result of rendering other elements (e.g., child elements) so that smaller rectangular section of the element would render, the visible rectangle is substituted for the original element or the size and/or position (e.g., coordinates and/or dimensions) of the partly occluded element are reduced to the smaller rectangular section.
Some embodiments can include a system for maintaining both an unoptimized original hierarchy of views and an optimized hierarchy of views. For example, the applicant of the present disclosure has developed a “React” JavaScript® library for building user interfaces. The React system enables a user (“developer” or “programmer”) to compose reusable user interface components and subcomponents to render views. When a page or a part of a page needs to be rendered (e.g., when it is initially displayed or updated), React returns a tree of components that will eventually be rendered (for example, to HTML elements like <div> or <span>, or to native display elements like Image or Text elements, or, e.g., MKMapView in iOS® or ImageView in Android™). React creates a data structure mirroring the original hierarchy of views, computes any differences resulting from a change to the displayed views, and then updates the displayed hierarchy efficiently. This enables a programmer to write program code as if the entire page is rendered on each change, thereby simplifying programming because the program code does not need to track which components are not visible, while the React libraries only render subcomponents that actually change. React computes the minimal set of changes necessary to keep the displayed hierarchy of views up-to-date. Although many embodiments are described herein in the context of React, embodiments are not limited to React or to rendering only updated views as determined by React libraries. Various embodiments enable optimization of views rendered for display in various computing environments, such as an application in an Android™ device. Embodiments operating in some or all of the ways described herein can reduce the number of rendering commands related to overdrawn areas, resulting in reduced rendering load, faster display rendering, and lower memory usage.
The following description provides certain specific details of the illustrated examples. One skilled in the relevant art will understand, however, that embodiments can be practiced without many of these details. Likewise, one skilled in the relevant art will also understand that the present disclosure can include many other obvious features not described in detail herein. Additionally, some well-known structures or functions may not be shown or described in detail below, to avoid unnecessarily obscuring the relevant descriptions of the various examples.
The processing component 130 is connected to memory 140, which can include a combination of temporary and/or permanent storage, and both read-only memory (ROM) and writable memory (e.g., random access memory or RAM, CPU registers, and on-chip cache memories), writable non-volatile memory such as flash memory or other solid-state memory, hard drives, removable media, magnetically or optically readable discs and/or tapes, nanotechnology memory, synthetic biological memory, and so forth. A memory is not a propagating signal divorced from underlying hardware; thus, a memory and a computer-readable storage medium do not refer to a transitory propagating signal per se. The memory 140 includes data storage that contains programs, software, and information, such as an operating system 142, application programs 144, and data 146. Computer system 100 operating systems 142 can include, for example, Windows®, Linux®, Android™, iOS®, and/or an embedded real-time operating system. The application programs 144 and data 146 can include software and databases—including data structures, database records, other data tables, etc.—configured to control computer system 100 components, process information (to, e.g., reduce overdraw in graphical view data), communicate and exchange data and information with remote computers and other devices, etc. In a distributed computing environment, program modules and data can be located in both local and remote memory storage devices.
In some embodiments, the memory 140 includes program memory that contains programs and software, and data memory that includes configuration data, settings, preferences, files, documents, etc. that may be accessed by instructions from the program memory or by a component of the computing system 100. Program memory can include modules of the operating system 142 and application programs 144, such as a view management system and view rendering software for displaying and modifying views according to the present disclosure, and communication software for transmitting and receiving data by various channels and protocols via the communication components 150.
The computer system 100 can include input components 110 that receive input from user interactions and provide input to the processor 130, typically mediated by a hardware controller that interprets the raw signals received from the input device and communicates the information to the processor 130 using a known communication protocol. Examples of an input component 110 include a keyboard 112 (with physical or virtual keys), a pointing device (such as a mouse 114, joystick, dial, or eye tracking device), a touchscreen 115 that detects contact events (e.g., when it is touched by a user), a microphone 116 that receives audio input (e.g., for systems implementing speech recognition as a method of input by the user), and a camera 118 for photograph and/or video capture. The computer system 100 can also include various other input components 110 such as GPS or other location determination sensors, motion sensors, wearable input devices with accelerometers (e.g. wearable glove-type or head-mounted input devices), biometric sensors (e.g., a fingerprint sensor), light sensors (e.g., an infrared sensor), card readers (e.g., a magnetic stripe reader or a memory card reader), and so on.
The processor 130 can also be connected to one or more various output components 120, e.g., directly or via a hardware controller. The output devices can include a display 122 on which text and graphics are displayed. The display 122 can be, for example, an LCD, LED, or OLED display screen (such as a desktop computer screen, handheld device screen, or television screen), an e-ink display, a projected display (such as a heads-up display device), and/or a display integrated with a touchscreen 115 that serves as an input device as well as an output device that provides graphical and textual visual feedback to the user. The output devices can also include a speaker 124 for playing audio signals, haptic feedback devices for tactile output such as vibration, etc. In some implementations, the speaker 124 and the microphone 116 are implemented by a combined audio input-output device.
In the illustrated embodiment, the computer system 100 further includes one or more communication components 150. The communication components can include, for example, a wired network connection 152 (e.g., one or more of an Ethernet port, cable modem, FireWire cable, Lightning connector, universal serial bus (USB) port, etc.) and/or a wireless transceiver 154 (e.g., one or more of a Wi-Fi transceiver; Bluetooth transceiver; near-field communication (NFC) device; wireless modem or cellular radio utilizing GSM, CDMA, 3G and/or 4G technologies; etc.). The communication components 150 are suitable for communication between the computer system 100 and other local and/or remote computing devices, directly via a wired or wireless peer-to-peer connection and/or indirectly via a communication link and networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like (which can include the Internet, a public or private intranet, a local or extended Wi-Fi network, cell towers, the plain old telephone system (POTS), etc.). The computer system 100 further includes power 260, which can include battery power and/or facility power for operation of the various electrical components associated with the computer system 100.
In the illustrated embodiment, the view manager component 304 is connected to an overlap subtraction component 306. The overlap subtraction component 306 can identify portions of a view (e.g., a background view) that intersect and are overlapped by another view (e.g., a foreground view). If the overlapping foreground view is opaque and thus occludes the overlapped background view, the overlap subtraction component 306 can subtract the dimensions of the overlapping or occluding portion from the background view. For example, the overlap subtraction component 306 can record in an overlap subtraction data structure, such as in a record associated with the background view, coordinates representing portions of the background view that are occluded on the display. In some embodiments, the overlap subtraction component 306 and/or a visible area computation component 308 can maintain a lightweight virtual representation of the occluded background view paralleling the unmodified view. The visible area computation component 308 is configured to determine whether, after subtraction of overlapping regions, the background view would cause anything to be displayed on the screen. In some embodiments, the visible area computation component 308 makes a binary determination of whether the overlapped element or view is completely occluded. In some embodiments, the visible area computation component 308 determines the remaining visible area, dimensions, and/or shape of the overlapped element or view. In some embodiments, the visible area computation component 308 is configured to determine whether the visible remainder portion of the view is a simple rectangle, as opposed to a more complex or irregular shape, to simplify resizing or other modification of the view by a view property modification component 310. In some embodiments, the visible area computation component 308 determines the smallest bounding rectangle in which a partly overlapped view can be fit, enabling the partially overdrawn view to be reduced in size to such a smaller rectangle.
Using the view information from the visible area computation component 308, a view property modification component 310 modifies the overdrawn view. In some embodiments, the view property modification component 310 sets a flag designating a view that is completely occluded as not to be drawn. In some embodiments, the view property modification component 310 omits such a completely occluded view from a hierarchy of views to be rendered. In some embodiments, the view property modification component 310 un-sets or otherwise removes a background property (e.g., an image or color) so that the view no longer directs a rendering engine to render that property. In some embodiments, the view property modification component 310 modifies a size property of the view (e.g., reducing the size of a partly occluded view to the minimum size needed to display the remaining visible portion of the view). By modifying properties of the view to avoid drawing overlapped regions that will not ultimately be displayed, memory resources that would be allocated to the view are saved, and the depth of the display hierarchy can be reduced in comparison to the original hierarchy of overlapping views. A display component 312 renders the modified views (e.g., via a native application programming interface (API) for rendering views, or a web browser XML DOM rendering engine) and displays the reduced-overdraw rendered views 314 after the modification of views to reduce overdraw by the view property modification component 310.
In the example illustrated in
In some cases, removing all overdrawn regions regardless of size, location, and/or dimension—including, e.g., scattered variously shaped elements—might end up creating more regions and causing more polygons to be rendered. In some embodiments, an automatic overdraw reduction system detects whether a background view is only partially occluded, and determines whether subtracting the overlying element would simplify the rendering of the overdrawn element. If not, the system can keep the overdrawn view without removing or modifying its properties in a manner that would increase the amount and/or complexity of the rendering load. In the illustrated example, the square view 506 does not affect the portion of the rectangular view 504 that is rendered, because subtracting the square view 506 from the rectangular view 504 does not reduce the rectangular view 504 to a smaller rectangular region.
In block 604, the computer system iterates through each child element of the background parent view to determine whether the parent view is partly or completely overdrawn by its children. In various embodiments, iterating through each child element includes fully traversing a tree or other data structure of the parent view's children (checking its children's children, and so on), e.g., via breadth-first search, depth-first search, recursion, etc. For example, a child of the parent view might be a container element that does not have a background color or image, but that contains its own children that do have backgrounds set. In some embodiments, iterating through child elements includes detecting overlapping views or elements that are not direct children, such as sibling elements within a common parent view (e.g., overlapping windows on a tablet screen).
In decision block 606, the computer system determines, for a particular child element selected in block 604, whether the child element is opaque. If the child element is not opaque (e.g., transparent as a result of not having a background image or color set, or translucent with partial opacity), then the child element does not occlude the portion of the parent view behind the child element, so the process continues iterating through child elements in block 620. If, however, the child element is opaque, then the process continues in block 608.
In block 608, the computer system determines the dimensions of the overlapping opaque portion of the child view that occludes the parent view, and subtracts those dimensions from the parent view (or, e.g., from a representation of the parent view to be substituted for the full parent view for rendering purposes). In block 610, the computer system computes the remaining portion of the parent view to be rendered after subtraction of the child view. For example, the subtraction of one or more child views may result in an empty parent view, in which no portion of the parent view will be rendered.
In decision block 612, if the parent view is completely occluded, such that no portion of the parent view remains to be rendered, the computer system determines that the parent view can be excluded from rendering, and the process continues in block 614. In block 614, the computer system removes the parent view background. In some embodiments, the system (e.g., the view property modification component 310 of
Returning to decision block 612, if the parent view is not completely occluded, then the process continues in decision block 616. In decision block 616, the computer system determines whether the rest of the parent view is a simple rectangle. For example, if the child element covers a corner of the parent element, leaving a Utah-shaped region of the parent element visible, or a box within the parent element, leaving a doughnut-shaped region of the parent element visible (e.g., element 504 of
In block 618, the computer system reduces the size of the parent view to the size of the non-occluded rectangle remaining to be rendered. For example, as illustrated in
In decision block 620, if the system has not iterated through all children of the parent view, then the process continues in block 604. Once all children have been iterated through, then overdrawn regions of the parent view have been removed if possible, reducing the computing system's rendering load, and the process continues in block 622. In block 622, the computer system renders the remainder of the parent view, which may be, e.g., the entire parent view or a smaller rectangular region. After block 622, the process ends.
While processes or blocks are presented in a given order herein, alternative implementations can perform process blocks in a different order, and some components or blocks can be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or sub combinations. Each of these components or blocks can be implemented in a variety of different ways. Also, while components or blocks are at times shown in series, these components or blocks can instead be performed or implemented in parallel, or can be performed at different times.
Embodiments operating in some or all of the ways described above provide a page rendering infrastructure that reduces the rendering of overdrawn view regions, reducing memory and processor utilization requirements and improving page or frame display speed.
From the foregoing, it will be appreciated that specific embodiments of the disclosure have been described herein for purposes of illustration, but that various modifications may be made without deviating from the spirit and scope of the various embodiments of the disclosure. Further, while various advantages associated with certain embodiments of the disclosure have been described above in the context of those embodiments, other embodiments may also exhibit such advantages, and not all embodiments need necessarily exhibit such advantages to fall within the scope of the disclosure. Accordingly, the disclosure is not limited except as by the appended claims.