Mobile devices that can display video are becoming extremely popular. Microsoft Corporation, the assignee of the present application, makes mobile devices with such capabilities, an example of which is the Windows Phone® mobile phone environment. Many companies, including Microsoft Corporation, make other portable devices that provide the capability of displaying video content. A characteristic of mobile devices with such capability is often a small screen size and limited battery life.
In spite of those and other inherent limitations, consumers typically demand that such devices be capable of displaying video content in a form and with related content without compromising battery life, in a fashion similar to that possible with devices having greater processor capabilities and longer battery life.
One aspect of the subject matter discussed herein is a method for reproducing continuous video content on a mobile phone LCD display by rendering plural source video textures as consecutive surfaces on the display. The method comprises (a) determining if a hardware scaler module is capable of rendering the particular surface, (b) rendering the surface using a general purpose graphical processing unit (GPU) if the response to the determining step (a) is negative, (c) determining if the particular surface is to be rendered with one or more additional images derived from a source other than a source video texture, (d) using the hardware scaler module to render the surface with the source video texture and any additional images if it is capable of doing so, and (e) repeating steps (a) through (d) for the next consecutive surface. As used herein, “continuous” video content refers to the successive display of frames of video data one after the other, typically at uniform intervals to reproduce a predetermined amount of the video data. A common real-time reproducing frequency is 60 frames per second, with slow motion, reverse and fast forward reproduction at higher or lower frequencies as the case may be. A video session is an uninterrupted time period in which multiple video textures are reproduced consecutively on the LCD surface, although it will be understood that a video session can involve the display of textures in an order different from that in which they were created.
It is preferable to use the hardware scaler to render surfaces on the LCD because it uses less battery power than the GPU. Efficient use of battery power is realized here by providing the capability of dynamically switching between the hardware scaler and the GPU for rendering the surfaces on a surface-by-surface basis, if necessary. Battery power can be further conserved by using for a next consecutive surface unchanged parts of a previous surface that would otherwise have to be generated again by the GPU for the next consecutive surface.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The objects of the subject matter discussed herein will be better understood from the detailed description of embodiments which follows below, when taken in conjunction with the accompanying drawings, in which like numerals and letters refer to like features throughout. The following is a brief identification of the drawing figures used in the accompanying detailed description.
One skilled in the art will readily understand that the drawings are schematic in many respects, but nevertheless will find them sufficient, when taken with the detailed description that follows, to make and use the claimed subject matter.
The processor component 100 further includes a web browser module 106 that is a particular type of executable program under the control of the operating system module 102, that allows a user of the mobile device to access or otherwise navigate to websites and download files. Access to websites on the Internet can be gained through well-known protocols embodied in firmware and/or software included in the web browser module 106. A typical such protocol is commonly known as Wi-Fi, but there is no limitation on the manner in which the device might access content from remote locations, including wired connections conforming to the well known USB standard or by the use of a portable memory device physically plugged into the device, just to name some examples. In any event, relevant to the present embodiment, the web browser module or other content source is operable to download video content to the device 10. In a typical arrangement the video content will be stored temporarily in the storage module 104 prior to further processing and display as explained in more detail further below. Video content can also be captured by a video recorder module 108 that is included in the mobile device 10 and is under the control of the operating system module 104. The mobile device has controls (not shown) by which a user can activate a video recording device included in the video recorder module 108. Typically, the video recorder module 108 will include features such as a zoom lens and other video recording controls operable by the user. Video content from whatever source derived is typically captured as a series of textures that are stored as blocks or frames of video data in the temporary storage module 104.
In the embodiment depicted, the mobile device 10 also includes a telephone module 110 that is under the control of the user via the operating system module 102. The telephone module includes the necessary circuitry and other components for connecting to cellular telephone networks in a conventional manner well known to those skilled in the art. The telephone module 110 is also capable of interacting with the web browser module 106 to download content from the Internet by various conventional protocols. A display component 112 provides a user interface by which information is conveyed to the user of the device. The display component in one embodiment is an LCD (liquid crystal display) that displays a touch screen through which the user can input commands to the operating system module 102. Such touch screen displays are also well known to those skilled in the art, who will be able to implement an LCD or other display as described herein without further explanation. The mobile device may also have other input devices such as conventional mechanical-electrical buttons and/or toggle switches (not shown in
It will be appreciated that the components of the device 10 thus far described are not exhaustive of the components that can be incorporated into a mobile device suitable for performing the video composing techniques described herein. For example, the device would typically also include a still photograph function and would also include an email application. It could further include a mapping function that enables the user to determine his or her position by displaying on the display component 112 a position indicator superimposed on a street map. The device will also typically include a module that enables a USB port for functions already discussed, as well as others, such as enabling the device to be physically connected to another electronic device to exchange information with such device (sometimes referred to as “synching”) and/or to an electrical outlet for recharging the battery included in the battery module 114.
As used in this description, the terms “component,” “module,” “system,” “apparatus,” “interface,” “unit” or the like are generally intended to refer to a computer-related entity or entities, either hardware, a combination of hardware and software, software, or software in execution, unless the context clearly indicates otherwise. For example, such a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer (device) and/or distributed between two or more computers (devices). Nor does the schematic depiction in the manner used herein of modules, components or units for performing various functions imply that such modules, components or units are physically separate or comprise discrete entities within a device for performing the methods and embodying the systems described herein. In other words, these depictions are not meant necessarily to represent discrete hardware entities, but rather as functional components that can be realized by one skilled in the art in any suitable fashion using hardware, software, or firmware in accordance with the description herein.
Further, a “computer storage medium” as used herein can be a volatile or non-volatile, removable or non-removable medium implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that now exists or may become available in the future that can be used to store the desired information and which can be accessed by a computer.
The mobile device 10 described here is meant to be only one example of an electronic device for effecting the video composition methods described herein. It is intended that “electronic device” be considered broadly as including any such device (or any physical or logical element of another device, either standing alone or included in still other devices) that is configured for communication via one or more communication networks such as those described herein and that is responsive to user inputs. While the methods described herein have particular utility when applied to battery-powered handheld mobile devices capable of performing some or all of the functions (or more) described above, those skilled in the art will immediately recognize that all manner of electronic devices may be adaptable to effect the methods to be described, examples of which electronic devices could include, but would not be limited to, mobile phones, personal digital assistants, smart phones, laptop and desktop computer systems of any configuration or implementation, personal media players, image or video capture/playback devices, devices temporarily or permanently mounted in transportation equipment such as planes, trains, or wheeled vehicles, set-top boxes, game consoles, stereos, digital video recorders/players, and televisions.
Returning now
As noted above, the source video data to be displayed (rendered) on the display component 112 is organized into blocks of data, each of which is rendered as a surface of the LCD display component 112. One of the functions of the video rendering engine module 116 is to organize digital video data, captured in one of the ways discussed above, for example, into addressing data for activating the rows and columns of the LCD display electrodes so as to render one such block of video data (also sometimes referred to as a “texture”) onto a surface of the LCD. By rendering such surfaces of the LCD display one after the other in succession in a video session, the video data is recreated on the display.
However, the source texture data may not be in a form or obtained in a manner that readily permits generation of surfaces for rendering on the LCD. For example, digital video data typically comprises a two-dimensional matrix of pixels (“picture elements”), each of which includes a sufficient number of bits to define the characteristics of the texture at the point represented by a particular pixel. Typical resolutions for an LCD display component on a so-called smartphone mobile device are from 240×320 pixels to 640×960 pixels, with a common resolution being 480×800 pixels. Usually, a user will desire that the source texture be matched to the LCD screen surface resolution, which might require scaling the source texture either up or down. In addition, many source textures are encoded in YUV color space, while most LCDs render images in RGB color space. Conversion algorithms are well known, but the source texture will have to be YUV-RGB converted before the image can be rendered for display on the LCD display component 112.
The hardware scaler module 118 and the GPU 120 perform operations on a source texture that enable it to be rendered as a surface on the LCD along with other items. Taking the hardware scaler module 116 first, it is typically designed to perform various predetermined source texture manipulations. The GPU module 120 can perform all of the functions of the hardware scaler module, but the latter performs those particular functions for which it designed using less power than a typical graphical processing unit. However, by the same token a typical general purpose GPU can perform functions that the hardware scaler cannot. For example, a typical MDP chipset is particularly efficient in terms of power usage for converting from YUV color space to RGB color space and for scaling a given source texture to the resolution of the particular LCD display screen in use. However, unlike most GPUs, a typical MDP hardware scaler chipset now in use usually cannot stretch a given texture to more than eight times its original resolution, or shrink it to less than one-fourth of its original resolution. It also cannot process textures smaller than 64×64 pixels, and can rotate images only in 90° increments. In addition, it cannot provide images that appear partially or wholly transparent so as to be able to display one image that appears to be on top of another while maintaining some visibility of the “bottom” image. The transparency of an image is typically termed “alpha” (α), with values between 0 (opaque) to 100 (totally transparent, that is, not visible). An MDP chipset can normally only render images with α=0.
Finally, of relevance to the methods and systems of the present description, an MDP chipset of the type typically used in a mobile device like that discussed herein cannot generate video images except for filling with black video data areas of the display other than the video texture. That is, if the source texture has an aspect ratio (width divided by height) that is different from the LCD surface aspect ratio, the texture must either be stretched or shrunk to match the LCD aspect ratio, or be displayed with borders (sometimes referred to as a “letterbox” format). A typical MDP chipset can only render these border areas in black. For purposes of this discussion, a display mode using the hardware scaler unit to render a surface comprising only source video content rendered by the MDP hardware scaler module is in this description referred to as the “secondary only optimized” mode. This display mode is discussed in more detail further below in connection with
The flowchart in
For purposes of this description a particular block or set of video data to be rendered on the LCD surface is considered as one or more generalized “sprites,” which are processed in turn for each LCD surface being composed. A step S104 determines if a sprite is a particular subset of sprites referred to herein as VideoSprites. In a typical implementation, a VideoSprite is a sprite that comprises the source video texture or the information used to render border areas for displaying a letterbox format as discussed above. Other sprites, that is, non-VideoSprites are processed differently, as will be discussed. Typically, the data defining a sprite in some fashion further identifies it as a Video Sprite. One example of a sprite that is not a VideoSprite is an image generated by the GPU and displayed on the LCD as video transport controls such as “Play,” “Pause,” Fast Forward,” “Reverse,” “Full Screen,” and the like. (See
The process depicted in
The next step is a step S114 in which the video rendering engine module determines if the VideoSprite is to be rendered as a simple rectangle on the LCD surface. For purposes of the present discussion, a “simple rectangle” is a VideoSprite that points to LCD screen coordinates that define a rectangle oriented at 0°, 90°, 180°, or 270° relative to the LCD screen pixels, and that does not contain color gradients. If the VideoSprite does not represent such a simple rectangle, it is generated by the GPU module for rendering as a primary image. The process then proceeds to the step S108, which sets the SecondaryState flag to “false.” If the VideoSprite is a simple rectangle, the process proceeds to a step S116 where it is determined whether or not the VideoSprite includes video content. If the VideoSprite does not include video content, that is, is not the source video texture but blank video data to be rendered as a black area on the LCD surface by the hardware scaler module, the process goes to the step 108, where the SecondaryState flag is set to “false.”
Next, if the step S116 determines that a VideoSprite has video content (that is, that it comprises source video texture), then the process proceeds to a step S118. Here it is determined if the VideoSprite representing the source video content can be rendered by the hardware scaler module 118. As noted above, the hardware scaler module has limited capabilities vis-à-vis the GPU module 120. For example, one conventional MDP chipset embodying the hardware scaler module as discussed above cannot stretch a given texture more than a predetermined amount (eight times in the present example), or shrink it to less than a predetermined fraction of its original resolution (one-fourth in the present example), and it cannot process source video textures smaller than a certain size (64 pixels×64 pixels in the present example). In addition, it can normally only render images with α=0 (opaque). If the VideoSprite cannot be rendered by the MDP hardware scaler chipset as requested by the video rendering engine module, the step S118 proceeds to the step S120, where the ScalerState flag is set to “false,” and then to the step S108 where the SecondaryState flag is also set to “false.” If, however, the source video texture can be rendered by the hardware scaler module 118, the process proceeds from the step S118 to the step S122, where the ScalerState flag is set to “true” The process continues to compose the LCD surface in accordance with the flowchart in
In a step S124 the ScalerState flag is checked. If it is not “true,” it means that the MDP hardware scaler is not capable of rendering the source video content (see the step S120), or any of the rest of the surface either (since the default state of the ScalerState flag is “false,” as per the step S102). After the GPU renders the current surface, the process returns to the step S100 and renders the next surface. If the ScalerState flag is “true” in the step 124, it means that the MDP hardware scaler module is capable of rendering the surface, and the process goes to a step S128, where the SecondaryState flag is checked (otherwise, step S126 is performed). If it is “true,” it means that only the source video texture is being rendered, and the process proceeds to a step S130 in which the hardware scaler module renders the surface in a secondary only optimized mode (see
Further details of the process thus far described are better understood by referring to
After the current surface is rendered either in the step S130 or the step S132, the process could return to the step S100 and render the next surface from scratch, as it were. However, if the primary image (in the examples mentioned above, text such as a title caption as depicted in
The only determination that needs to be made to enable the hardware scaler module to compose this surface is to confirm in a step S204 that the new source video texture meets the MDP chipset requirements (the step S204 corresponds generally to the steps S114 and S118 in
As described, there are essentially four manners in which the present method and apparatus can compose each of a succession of video surfaces. One is to render a new primary image and a new secondary image. For example, if transport controls are displayed, or they are to appear for the first time, they must be redrawn or initially drawn in rendering the next consecutive surface. This new surface cannot be rendered using the flowchart in
The hardware scaler module 118 is typically capable of rendering video content from more than one source video. For example, this could be done by rendering the different video textures as a split screen surface on the LCD display, or in a “picture-in-picture” format wherein one or more additional source video textures are rendered in small boxes in a surface comprising a main source video. The methods and apparatus described above can utilize that capability as well. That is, since a typical hardware scaler module is capable of rendering multiple VideoSprites (five in the above described embodiment), more than one of the VideoSprites can include video content, as opposed to only one of them as described above. In effect, one or more of the black VideoSprites generated by the hardware scaler module can be video content instead. One skilled in the art will be readily able to make the necessary modifications to the flowcharts discussed above to effect this alternate embodiment.
Unless specifically stated, the methods described herein are not constrained to a particular order or sequence. In addition, some of the described method steps can occur or be performed concurrently. Further, the word “example” is used herein simply to describe one manner of implementation. Such an implementation is not to be construed as the only manner of implementing any particular feature of the subject matter discussed herein. Also, functions described herein as being performed by computer programs are not limited to implementation by any specific embodiments of such program.
Although the subject matter herein has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter of the appended claims is not limited to the specific features or acts described above. Rather, such features and acts are disclosed as sample forms of corresponding subject matter covered by the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
6470051 | Campisano et al. | Oct 2002 | B1 |
6573905 | MacInnis et al. | Jun 2003 | B1 |
6828982 | Lee | Dec 2004 | B2 |
6983017 | Chen et al. | Jan 2006 | B2 |
7455232 | Epshteyn | Nov 2008 | B2 |
7548245 | Evans et al. | Jun 2009 | B2 |
7898545 | Alben et al. | Mar 2011 | B1 |
20060117356 | Jojic et al. | Jun 2006 | A1 |
20080143749 | Weybrew et al. | Jun 2008 | A1 |
20080284798 | Weybrew et al. | Nov 2008 | A1 |
20090147854 | Dane et al. | Jun 2009 | A1 |
20090184977 | Weybrew et al. | Jul 2009 | A1 |
Entry |
---|
“Samsungs New Dual-Core Mobile Processor”, Retrieved at <<http://phill.co/phone-reviews/sannsungs-new-dual-core-mobile-processor>>, Retrieved Date: Feb. 3, 2011, pp. 7. |
“Shared Surface Hardware-Sensitive Composited Video”, U.S. Appl. No. 12/912,941, filed Oct. 27, 2010, pp. 27. |
Number | Date | Country | |
---|---|---|---|
20120313954 A1 | Dec 2012 | US |