Patent Application
20090300489
In various embodiments, the present invention relates to display of data in an electronic device, and more particularly to providing selective access to a frame buffer to improve efficiency and reduce artifacts.
It is well known to provide a buffer to store display data for display in a windowing system. A windowing system is a software system that permits application sharing of a display. One example of such a windowing system is a desktop comprising an electronic representation of a workspace in which windows are spawned for specific applications to display output.
A frame buffer is typically used by a windowing system to temporarily store display data used to render video displayed in a window. An application requests a location in memory to temporarily store its display data depending on a size and location of an associated window. The size and location can both be modified by user input from, for example, a user dragging a window across a desktop and resizing for a better view. Certain applications, such as video play-back applications or video games that are external to the display system, can be slow in adjusting how display data is stored in response to user interactions. Such applications often assume they have full and direct access to the frame buffer, and therefore are not aware of other considerations that may affect their display data as it is to be shown on the screen. Accordingly, display data from such applications may be written to a stale location or without regard to a new visual element layered on top, resulting in display artifacts. The display artifacts are visually undesirable.
In addition, direct access to the frame buffer can cause difficulties in handling occlusions and compositing. For example, if a visual element is dragged over the window, it may be necessary to occlude the window and/or composite it with the overlaying visual element. But if the application generating the window is given direct access to the frame buffer, it may erroneously write display data over space occupied by the visual element. In another example, when dragging a window, the application may continue to write display data over space that is no longer occupied by the window.
What is needed is a method to selectively redirect an application to an external frame buffer when such redirection is beneficial to properly handle occlusions and/or other interactions between a display window and other visual elements being displayed. In such situations, a windowing system can then perform separate processing before the final result is sent to the frame buffer. What is further needed is a method that takes advantage of improved efficiency that results from direct access to a frame buffer when no occlusions and/or interactions need to be processed.
In various embodiments, a method, system, and computer program are provided for displaying data in a window or other display area by selectively providing access to a frame buffer. When a request for a destination location for display data is received, a determination is made as to whether the display data will require further processing prior to display. If further processing is needed, a location of an external application buffer is provided. If no further processing is needed, a frame buffer is provided. Thus, improved efficiency is achieved, while still providing functionality for processing occlusions and/or composited sections of the display data.
For example, in one embodiment a display window can be partially occluded (clipped) or composited with a visual element such as a menu, icon, or other window. However, when there is no overlapping in the display, the frame buffer can be used.
In one embodiment, advantageously, when further processing is needed, the application is not given direct access to the frame buffer, and the associated display data can be processed as needed. This reduces artifacts and ensures that compositing and/or occlusions are handled properly. Improved efficiency resulting from direct access to the frame buffer is maintained when such further processing is not needed.
The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention. One skilled in the art will recognize that the particular embodiments illustrated in the drawings are merely exemplary, and are not intended to limit the scope of the present invention.
For purposes of the description provided herein, the following terms are defined as follows:
“Desktop” refers to an electronic workspace used to display windows and other visual elements such as menus and icons.
“Display window” (or “window”) refers to an on-screen area for displaying data. An example is a window used to display video from an external application or other types of graphics such as video game display, documents, and the like.
“Overlapping” refers to a display window that is intersects with another visual element in such a manner that requires occlusion and/or compositing.
“Non-overlapping” refers to a display window that does not intersect with another visual element, or that intersects with another visual element in a manner that does not require alteration of the display window's content (for example, if the display window is topmost and has no translucency).
“Frame buffer” refers to an area of memory that is used for receive display data for an output device.
“Application buffer” refers to another area of memory that is used by an application to receive display data.
In
In one embodiment, the memory 110 receives data for display on display device 122. Such data can include, for example, video frames. Such display data can originate from within system 100 (e.g., stored on the hard drive 140) or from outside the system 100 (e.g., downloaded from the Internet or from an external device coupled to the I/O controller 150). The display module 112 processes the display data as described in more detail below with respect to
Referring to
In various embodiments, the application 210 can be a third-party application, or an application that is integrated with an operating system. The desktop display module 220 manages the composition of a desktop and user interactions with visual elements such as windows, icons, and menus. The desktop display module 220 tasks a windowing module 215 to display applications within windows on a desktop. The windowing module 215 can spawn a new window to provide a contained environment for an application. Since several windows can be displayed on a desktop, along with other visual elements, there are times when the location of one window overlaps with the location of another window or visual element. As a result, the desktop display module 220 can selectively process display data based on whether there is overlap, on a frame-by-frame basis as described in more detail below with respect to
The invention is described herein in the context of displaying visual elements in a desktop metaphor, as is common in many window-based user interfaces. However, one skilled in the art will recognize that this description is merely illustrative of a particular embodiment, and that the present invention can be implemented in other contexts as well.
The frame buffer 230 can be a single buffer or a collection of partitioned buffers (e.g., a buffer for each window, a buffer for each video frame, or the like). In one embodiment, buffers exists in RGB configuration in which a dedicated buffer for each video color of red, green, and blue. In another embodiment, one buffer receives an RGB frame and another buffer receives a YUV frame, and the frames are combined for output. The frame buffer 230 receives display data for the display device 122.
In one embodiment, as described in more detail below, the frame buffer 230 receives display data (such as frames of video) when the associated window does not overlap with other visual elements. Conversely, the external application buffer 240 receives display data when at least a portion of the associated window is overlapping, thus allowing for compositing or clipping prior to final display.
Referring now to
In one embodiment, the arbitration 701 takes place for each frame of display data generated by each application 210. In response to the application 210 requesting a destination for display, the system of the present invention provides a memory address representing either an external application buffer 240 or the frame buffer 230. In the example of
In various embodiments, once the application 210 is given a memory address to use as a destination for display data, the application 210 writes its display data to the specified location. A windowing module 215 or other component performs compositing operations on data in the external application buffers 240 to generate composited data. The composited data is written to the frame buffer 230. The frame buffer 230 can therefore contain display data written directly by one or more applications 210, display data resulting from compositing operations 702, or any combination thereof. The contents of the frame buffer 230 are then output on the display device 122.
In some cases, the desktop display module 220 provides a memory address corresponding to the external application buffer 240; in other cases, the desktop display module 220 provides the address of the frame buffer 230. In the latter case, either the entire frame buffer 230 or a portion thereof can be made available. In this manner, in step 320, the display data is either received at the frame buffer 230 or processed through the external application buffer 240, depending on whether compositing or occlusions are present.
More specifically, in some embodiments the destination address can be the frame buffer 230 when the display data does not require occlusion by or compositing with other visual elements. However, in the cases of occlusion or compositing, the destination address can be the external application buffer 240 so that further processing can take place before the display data is written to the frame buffer 230. The application 210 can store the destination address, where it can stay the same until the application 210 needs to write new display data. By preventing direct access to the frame buffer 230 when additional display processing is needed, the present invention in some embodiments ensures that the display data will be displayed properly including occlusions and/or compositing. By allowing direct access to the frame buffer 230 when no additional display processing is needed, the present invention provides improved efficiency when possible.
In one embodiment, the display window is displayed 330 on the display device 122. In one embodiment, this step includes displaying the data in a window of a user interface having a desktop metaphor. As discussed, the display module 112 and/or the display controller 120 perform various processing such as rendering, prior to output. The window size for containing displayed output can be allocated by the desktop display module 220.
A request is received 410 for a destination address of the frame buffer 230 to write display data such as a video frame. A determination 420 is made as to whether a display area, such as a window, is occluded (at least partially) by a visual element during display, and/or whether the display window overlaps a visual element in a manner that requires compositing (for example if the visual element and/or the display window is/are to be rendered translucently). In making this determination, a desktop layout can be generated using the size and location of each window being displayed. Preferably, the determination 420 is made in a highly efficient manner so as to provide the functionality of the present invention without interrupting or affecting the display of data in a perceivable manner. Further details are described below with regards to
Responsive to a display with occlusion and/or compositing, the external application buffer 240 is selected 430, so that the application 210 is not given access to the frame buffer 230. Rather, the display module 112 can control how the output video is displayed from the external application buffer 240. After the display module 112 properly renders the partially occluded (clipped) or composited display, in some embodiments, the resulting video data is then stored in the frame buffer 230.
If no occlusion and/or compositing is needed, the frame buffer 230 is selected 431.
The selected destination memory address is provided to the application 210 by a response 432 to the application's 210 request.
In step 440, the method repeats for each video frame or other quantum of display data.
If there is an intersection, it is determined whether the desktop either partially occludes (clips) the display window, in step 520, or whether the visual element is to be composited with the display window, in step 530 (for example, if either the visual element and/or the display window are to be displayed translucently). In one embodiment, occlusions and/or compositing can occur when at least a portion of a visual element and a display window each occupies the same pixel or segment.
If there is clipping or compositing, the display window overlaps the visual element 550, and an external application buffer 240 is selected. In one embodiment, the application 210 signals the display module 112 to indicate that it has completed writing data to the external application buffer 240, so that compositing or other processing can commence.
On the other hand, if there is neither clipping nor compositing, although there is an intersection, the display window does not overlap the visual element 540, and the frame buffer 230 is selected.
A desktop 600 can be displayed on devices as described herein. A user can interact with the desktop using any known input mechanism, including, for example, a mouse, a keyboard, a finger, a stylus, voice commands, and the like. In some cases, it may be useful to provide an on-screen cursor 615.
In one embodiment, the user can drag the display window 610 from one portion of the desktop 600 to another portion, as shown when comparing
Advantageously, by selectively redirecting the display data of the application 210 to the external application buffer 240 rather than the frame buffer 230, video artifacts can be reduced and improved performance can be achieved.
The present invention has been described in particular detail with respect to one possible embodiment. Those of skill in the art will appreciate that the invention may be practiced in other embodiments. First, the particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Further, the system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements, or entirely in software elements. Also, the particular division of functionality between the various system components described herein is merely exemplary, and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.
Reference herein to “one embodiment”, “an embodiment”, or to “one or more embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. Further, it is noted that instances of the phrase “in one embodiment” herein are not necessarily all referring to the same embodiment.
Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times, to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “displaying” or “determining” or the like, refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention can be embodied in software, firmware or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Further, the computers referred to herein may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
The algorithms and displays presented herein are not inherently related to any particular computer, virtualized system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references above to specific languages are provided for disclosure of enablement and best mode of the present invention.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised which do not depart from the scope of the present invention as described herein. In addition, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the claims.
