Patent Application
20130057567
1. Field of the Invention
This invention is related to the field of graphical information processing, more particularly, to displaying mirror images on multiple displays.
2. Description of the Related Art
Part of the operation of many computer systems, including portable digital devices such as mobile phones, notebook computers and the like is the use of some type of display device, such as a liquid crystal display (LCD), to display images, video information/streams, and data. Accordingly, these systems typically incorporate functionality for generating images and data, including video information, which are subsequently output to the display device. Such devices typically include video graphics circuitry to process images and video information for subsequent display.
In digital imaging, the smallest item of information in an image is called a “picture element”, more generally referred to as a “pixel”. For convenience, pixels are generally arranged in a regular two-dimensional grid. By using this arrangement, many common operations can be implemented by uniformly applying the same operation to each pixel independently. Since each pixel is an elemental part of a digital image, a greater number of pixels can provide a more accurate representation of the digital image. The intensity of each pixel can vary, and in color systems each pixel has typically three or four components such as red, green, blue, and black.
Most images and video information displayed on display devices such as LCD screens are interpreted as a succession of image frames, or frames for short. While generally a frame is one of the many still images that make up a complete moving picture or video stream, a frame can also be interpreted more broadly as simply a still image displayed on a digital (discrete, or progressive scan) display. A frame typically consists of a specified number of pixels according to the resolution of the image/video frame. Information associated with a frame typically consists of color values for every pixel to be displayed on the screen. Color values are commonly stored in 1-bit monochrome, 4-bit palletized, 8-bit palletized, 16-bit high color and 24-bit true color formats. An additional alpha channel is oftentimes used to retain information about pixel transparency. The color values can represent information corresponding to any one of a number of color spaces.
One color space is YPbPr, which is used in video electronics, and is commonly referred to as “component video”. YPbPr is the analog representation of the YCbCr color space, which is associated with digital video. The YPbPr color space and YCbCr color space are numerically equivalent, with scaling and offsets applied to color values in the YPbPr color space to obtain corresponding color values in the YCbCr color space. Color space conversion is the translation of the representation of a color value from one color space to another, and typically occurs in the context of converting an image that is represented in one color space to another color space, with the goal of making the translated image look as similar as possible to the original. For example, color values in the YPbPr color space are created from the corresponding gamma-adjusted color values in the RGB (red, green and blue) color space, using two defined constants KB and KR. In general, color video and/or image information may be separated into Chrominance (chroma or C for short, where Cb represents the blue-difference chroma component and Cr represents the red-difference chroma component) and Luminance (luma, or Y for short) information. Chrominance signals are used to convey the color information separately from the accompanying Luminance signal, which represents the “black-and-white” or achromatic portion of the image, also referred to as the “image information”.
In certain situations, there is a need to display the same images concurrently on multiple displays of a computer system. For example, a computing device may have an internal display, and may also include an interface to which an external display can be coupled. It may be desirable to couple an external display to the device even if the device already has an internal display, for example when giving a presentation—such as a software demonstration to an audience in a large room. The presenter may view the demonstration on the device's internal display while the audience views the demonstration on the external display. In making such a presentation, it is typically desirable for the two displays to show the same images at the same time (or at least such that differences between the two displays are not visually apparent). Achieving such a result, however, may require significant resources of the computing device. Such an allocation of resources may not make sense from a design standpoint, particularly where real estate is at a premium on the computing device (e.g., the computing device is a tablet or smart phone device) and the presentation feature described above is not frequently used. Further complicating the situation is the multiplicity of possible external displays of differing resolutions that may be attached to the computing device
Other corresponding issues related to the prior art will become apparent to one skilled in the art after comparing such prior art with the present invention as described herein.
In one set of embodiments, a video/image stream may be displayed, in mirror mode, on an internal display and an external display. To provide the stream to the external display, a two-wire display port interface to the external display may be supported on a thirty-pin connector on the device sourcing the video/image stream. In one embodiment, the device is an iPad™. For a certain screen resolution, for example a 2048×1536 screen resolution, there may not be enough bandwidth on the two-wire interface to transmit the full resolution image. In order to maintain a specified aspect ratio, for example a 4:3 aspect ratio (or a 3:4 aspect ratio, if the device is an iPad™ that is rotated) and the original image resolution, a color space conversion may be performed from an RGB color space (in which the video/image stream is sourced to the internal display) to the YCbCr color space, to allow for chroma subsampling.
By converting the stream into luminance and chrominance information, the stream may be encoded by implementing less resolution for chroma information than for luma information, taking advantage of the human visual system's lower acuity for color differences than for luminance. Since human vision has finer spatial sensitivity to luminance (“black and white”) differences than chromatic (color) differences, the chromatic information may be transmitted to the external display at a lower resolution, optimizing perceived detail at a particular bandwidth. In other words, the Y (Luma) component may be transmitted at full resolution, while the chroma (Cb and Cr) components may be scaled. Accordingly, there is no loss of image resolution, while some amount of color resolution is lost, but this loss of the color resolution isn't as perceptible as would be a loss of image resolution. This also avoids the need to retime frames within the system on chip (SOC) sourcing the video/image stream, and the horizontal and vertical dimensions of the image frame may be resized off-chip using an external scaler/rotator unit.
The following detailed description makes reference to the accompanying drawings, which are now briefly described.
While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean including, but not limited to.
Various units, circuits, or other components may be described as “configured to” perform a task or tasks. In such contexts, “configured to” is a broad recitation of structure generally meaning “having circuitry that” performs the task or tasks during operation. As such, the unit/circuit/component can be configured to perform the task even when the unit/circuit/component is not currently on. In general, the circuitry that forms the structure corresponding to “configured to” may include hardware circuits and/or memory storing program instructions executable to implement the operation. The memory can include volatile memory such as static or dynamic random access memory and/or nonvolatile memory such as optical or magnetic disk storage, flash memory, programmable read-only memories, etc. Similarly, various units/circuits/components may be described as performing a task or tasks, for convenience in the description. Such descriptions should be interpreted as including the phrase “configured to.” Reciting a unit/circuit/component that is configured to perform one or more tasks is expressly intended not to invoke 35 U.S.C. §112, paragraph six interpretation for that unit/circuit/component.
As shown in
In addition to display 120, computing device 110 includes an external interface 130 that may couple to an external or secondary display 160 via connection 150. Interface 130 may be any type of standard or proprietary interface, and may be wired or wireless. A given interface 130 can be understood to have a “data width” (e.g., a number of pins) dedicated to a specified amount of data the interface can transfer at a given point in time. Specifically, interface 130 may have a specified number of lines dedicated to transferring graphics (e.g. video/image) information to external display 160. Interface 130 may also be configured to provide data to other types of external devices that may also be coupled to computing device 110 via interface 130, in lieu of or in addition to external display 160. Connection 150 is a logical representation of the connection between device 110 and secondary display 160. In various embodiments, connection 150 may be wireless. In other embodiments, connection 150 may be wired, and may include one or more intervening hardware components, such as a vertical scaling unit discussed below. Like primary display 120, secondary display 160 may be any suitable type of device. In one embodiment, secondary display 160 is a high-definition TV (HDTV) compatible device.
Computing device 110 may include various structures (not depicted in
Processors within device 110 may implement any instruction set architecture, and may be configured to execute instructions defined in that instruction set architecture. The processors may employ any microarchitecture, including scalar, superscalar, pipelined, superpipelined, out of order, in order, speculative, non-speculative, etc., or combinations thereof. The processors may include circuitry, and optionally may implement microcoding techniques. The processors may include one or more L1 caches, as well one or more additional levels of cache between the processors and one or more memory controllers. Other embodiments may include multiple levels of caches in the processors, and still other embodiments may not include any caches between the processors and the memory controllers.
Memory controllers within device 110 may comprise any circuitry configured to interface to the various memory requestors (e.g. processors, graphics circuitry, etc.). Any sort of interconnect may be supported for such memory controllers. For example, a shared bus (or buses) may be used, or point-to-point interconnects may be used. Hierarchical connection of local interconnects to a global interconnect to the memory controller may be used. In one implementation, a memory controller may be multi-ported, with processors having a dedicated port, graphics circuitry having another dedicated port, etc.
Memory within device 110 may be any type of memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such as mDDR3, etc., and/or low power versions of the SDRAMs such as LPDDR2, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. One or more memory devices may be coupled onto a circuit board to form memory modules such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc. Alternatively, the devices may be mounted with a system on a chip in a chip-on-chip configuration, a package-on-package configuration, or a multi-chip module configuration.
Graphics controllers within device 110 may be configured to render objects to be displayed into a frame buffer in the memory. The graphics controller may include one or more graphics processors that may execute graphics software to perform a part or all of the graphics operation, and/or hardware acceleration of certain graphics operations. The amount of hardware acceleration and software implementation may vary from embodiment to embodiment. More specifically, referring to
In one set of embodiments, the output frames may be presented to the display controller through an asynchronous FIFO (First In First Out) buffer in display generation unit 210. The display controller may control the timing of the display through a Vertical Blanking Interval (VBI) signal that may be activated at the beginning of each vertical blanking interval. This signal may cause the graphics processor(s) to initialize (Restart) and start (Go) the processing for a frame (more specifically, for the pixels within the frame). Between initializing and starting, configuration parameters unique to that frame may be modified. Any parameters not modified may retain their value from the previous frame. As the pixels are processed and put into the output FIFO, the display controller may issue signals (referred to as pop signals) to remove the pixels at the display controller's clock frequency (indicated as VCLK). The pixels thus obtained may be queued up in the output FIFO at the clock rate (indicated as CLK) of the processing elements within display generation unit 210, and fetched by the display controller at the display controller's clock rate of VCLK.
In various embodiments, different structures within computing device 110 may be located within a system on a chip (SoC). In one implementation, device 110 includes integrated display 120, an SoC, memory, and interface 130, with the SoC coupled to the display, the memory, and the interface. Other embodiments may employ any amount of integrated and/or discrete implementations.
Computing device 110 may operate to display frames of data. Generally, a frame is data describing an image to be displayed. As mentioned above, a frame may include pixel data describing the pixels included in the frame (e.g. in terms of various color spaces, such as RGB or YCbCr), and may also include metadata such as an alpha value for blending. Static frames may be frames that are not part of a video sequence. Alternatively, video frames may be frames in a video sequence. Each frame in the video sequence may be displayed after the preceding frame, at a rate specified for the video sequence (e.g. 15-30 frames a second). Video frames may also be complete images, or may be compressed images that refer to other images in the sequence. If the frames are compressed, a video pipeline in device 110 may decompress the frames.
As also mentioned above, a display generation unit 210 within device 110 may be configured to read frame data from memory and to process the frame data to provide a stream of pixel values for display. The display generation unit may provide a variety of operations on the frame data (e.g. scaling, video processing for frames that are part of a video sequence, etc.). The unit may be configured as a display pipeline in some embodiments. Additionally, the display generation unit may be configured to blend multiple frames to produce an output frame. For example, in one embodiment, each frame pixel may have an associated alpha value indicating its opaqueness. The display generation unit may include one or more user interface blocks configured to fetch and process static frames (that is, frames that are not part of a video sequence) and one or more video pipelines configured to fetch and process frames that are part of a video sequence. The frames output by the user interface blocks may be blended with a video frame output by the video pipeline. In one embodiment, the display generation unit may be configured to provide the output pixel stream to pixel processing units (PPUs) within device 110.
Generally, a pixel value in a stream of pixel values may be a representation of a pixel to be displayed on a display coupled to device 110. The pixel value may include a one or more color space values. For example, in an RGB color space, the pixel value includes a red value, a green value, and a blue value. Each value may range from zero to 2N−1, and describes an intensity of the color for that pixel. Similarly, in the YCbCr color space, the pixel value includes a Y value, a Cr value, and a Cb value. The location of a pixel on the display may be inferred from the position of the corresponding pixel value in the pixel stream. For example, the pixel stream may be a series of rows of pixels, each row forming a line on the display screen. In a progressive-mode display, the lines are drawn in consecutive order and thus the next line in the pixel stream is immediately adjacent to the previous line. In an interlaced-mode display, consecutive passes over the display draw either the even or the odd lines, and thus the next line in the pixel stream skips one line from the previous line in the pixel stream. For brevity, the stream of pixel values may be referred to as a pixel stream, or a stream of pixels. Pixel processing units within device 110 may be configured to perform various pixel operations on the pixel stream and may provide the processed pixel stream to the respective physical interfaces (PHYs).
Generally, a pixel operation may be any operation that may be performed on a stream of pixels forming a line on a display. For example, pixel operations may include one or more of: color space conversions, backlight control, gamma correction, contrast ratio improvement, filtering, dithering, etc. The PHYs may generally include the circuitry that physically controls the corresponding displays. The PHYs may drive control signals that physically control the respective display panels in response to the pixel values. Thus, for example, a PHY for a display that is controlled by RGB signals may transmit voltages on the R, G, and B signals that correspond to the R, G, and B components of the pixel. There may also be a display clock that may be transmitted by the PHYs, or the display clock may be embedded in one of the control signals. Different PHYs for different displays may have clocks that are within different clock domains.
A “clock domain” refers to the circuitry that is controlled responsive to a given clock. Clocked storage devices such as latches, registers, flops, etc. may all be configured to launch and capture values responsive to the given clock, either directly or indirectly. That is, the clock received by a given clocked storage device may be the given clock or a clock that is derived from the given clock. On the other hand, clocked storage devices in a different clock domain launch/capture values responsive to a different clock that may not have a synchronous relationship to the given clock.
It is often desirable to use computing device 110 to make a presentation—for example, to an audience in a large room. In such a situation, the size of primary display 120 may be inadequate for audience members. To facilitate such presentations, secondary display 160 may be coupled to device 110 via interface 130 and connection 150. In this manner, the presenter may view the presentation on display 120 while the audience views the presentation on display 160. Such dual display becomes less useful, however, if images on the displays are not synchronized (that is, someone viewing the two images can visually discern image drift or other visual discrepancies). Stated another way, it is often desirable that the two images be displayed concurrently, such that when the presenter is describing a feature of the presentation appearing on display 120, this same feature is also appearing on display 160 at the same time. (As will be described further below, there may be some inherent phase difference between images on different display. As used herein, however, references to “synchronized,” “synchronous,” or “concurrent” display of images includes display of images on different displays that do not have visually discernable image drift.)
Concurrent display of images becomes more difficult when the internal display and external display have different resolutions (i.e., different number of pixels in the horizontal and vertical directions). One possible solution is to have different display generation units for each display. Such an approach has significant drawbacks. Consider a game developer who wishes to demonstrate a new video game using internal and external displays. If the video game is pushing the processing power of device 110, it may be a waste of processing power to have a second display generation unit running for the external display, when in effect it would be rendering the same image as for the first display generation unit. Thus, such a configuration might not allow the developer the ability to showcase the video game running at peak performance.
An alternative solution is the use of a “mirror mode” in which a single display generation unit is used to provide output (e.g., pixels) to displays 120 and 160. This solution involves fetching data from memory only a single time (as opposed to twice in the solution described above). In some embodiments of computing device 110, however, the use of mirror mode may still have shortcomings. In particular, in some instances, the data width of interface 130 may not provide sufficient bandwidth to concurrently display images on both displays. For example, interface 130 may be sufficient for many data transfer applications, but may not have enough pins to display video on an HDTV secondary display concurrently with the primary display. In order to facilitate concurrent display of images through such a connector, the data sent to interface 130 may be downscaled/compressed. However, compression can mean loss of image resolution, which may require a retiming of the frames before they are transmitted over interface 130. In one set of embodiments, a converter scaling unit may compress the image without loss of pixel resolution, thereby preventing the need to retime the frames before they are output over interface 130, as will be described next, with respect to
As described above with reference to
As described above, in some embodiments, the data width of interface 130 is less than that of an interface to primary display 120. In these situations, in order to effectuate display of images on secondary display 160 concurrently with display of images on primary display 120, interface 130 can be redesigned or the data passing through interface 130 may be compressed. Redesign of interface 130 may be problematic, particularly in situations in which the connector has been widely adopted over time.
In one embodiment, computing device 110 achieves concurrent display on external display 160 through bandwidth-limited interface 130 by scaling at least a portion of the data in between display generation unit 210 and interface 130. In the embodiment shown, a converter scaling unit 220 may perform color space compression, e.g. converting incoming pixel information represented in the RGB color space into pixel information represented in the YCbCr color space. The converter scaling unit 220 may subsequently downscale the chrominance information of the color converter pixels, thereby maintaining the geometric image resolution, while reducing the bandwidth of the data transmitted through interface 130. In other words, if the pixel information produced by display generation unit is not in a YCbCr color space format, (e.g. if it is in an RGB color space format), the pixel information may first be converted into the YCbCr color space format, and the chrominance information compressed as will be further described below. In embodiments where the pixel information is provided in the YCbCr color space format by display generation unit 210 to internal display 120, converter scaling unit 220 may operate on the received pixel information without requiring color space conversion. As one example, converter scaling unit 220 may receive 2048 pixels for a given line of a frame to be displayed on display 120, and there may be 1536 lines in a given frame. By compressing the chroma components of the pixel information and not the luma components, the same image resolution in terms of horizontal pixels by vertical pixels may be maintained for transmitting the pixel information through interface 130, thus not requiring a retiming of the frames to be transmitted over interface 130.
The implementation of
Note that in some embodiments, for example when the screen resolution of external display 160 is less than the resolution of internal display 120, horizontal and vertical scaling between interface 130 and external display 160 may be required. The scaling factor in such cases may also need to account for additional factors, such as a current orientation of computing device 110 (i.e., whether device 110 is in a portrait or landscape mode). While it may be possible to perform such scaling within device 110, it would again require retiming of the frames as they are transmitted through interface 130, and the additional complexity that such scaling circuit would entail is not warranted when considering the typical frequency of the use of mirror mode as compared to the additional hardware resources that would need to be allocated to perform the HVP scaling within device 110. In the embodiment shown in
HVP scaling is thus performed outside of device 110. In the embodiment shown, scaling and rotating unit 230 is a hardware device located within connection 150. In one embodiment, unit 230 is a dongle that couples to interface 130 and provides a connection (either wired or wireless) to external display 160. Alternate embodiments are possible. For example, unit 230 could be situated at the other end of connection 150, or even within external display 160. The configuration shown in
As discussed above, a problem may exist when a data width of interface 130 does not permit concurrent display of images on displays 120 and 160 (even leaving aside the differences in resolution). Chroma scaling unit 220 may operate to reduce the chroma channels by an amount sufficient to pass data through interface 130 at a rate that supports concurrent display of images (while leaving the luma channel uncompressed). In certain embodiments, unit 220 may thereby transmit an image having the same aspect ratio as that of the image displayed on display 120, which allows proportionately sized concurrent images to appear on displays 120 and 160 even when the resolution of display 160 differs from that of internal display 120.
In the example shown, an image displayed on display 120 at 2048×1536 pixels is ultimately downscaled to fit on a 1920×1080 display. In one embodiment, the scaling factor applied by unit 230 is based on whichever dimension (horizontal or vertical) needs the greatest amount of down-scaling. In
For certain implementations of computing device 110, the aspect ratio of display 120 may change. In one embodiment, the aspect ratio of display 120 may change based on the orientation of device 110. For example, device 110 may be configured such that if it is oriented (e.g., by the user) in a “landscape” mode (as in
The generation of vertical and horizontal control signals for display 160 (and also internal display 120) also need to be considered. Examples of such signals are shown with reference to
Vertical sync period 816 starts at the beginning of a frame. The vertical back porch period 818 starts at the end of vertical sync period 816 and lasts until the beginning of the first line of active pixel data (i.e., the beginning of vertical active period 812). The vertical front porch period 814 starts at the end of the last active line of pixel data and lasts until the beginning of the next frame (i.e. the beginning of the next vertical sync). Each of these periods may be defined as an integer multiple of the horizontal line time (reference numeral 854 in
Similarly, the horizontal blanking interval (HBI) 858 is the period between the last active pixel of one horizontal line and the first active pixel of the subsequent line, and is composed of a horizontal sync period 816, a horizontal back porch (HBP) period 868, and a horizontal front porch (HFP) period 864.
The horizontal sync period 816 starts at the beginning of a line. The horizontal back porch period 868 starts at the end of the horizontal sync period 816 and lasts until the first active pixel of the line (i.e., the beginning of horizontal active period 862—thus, for display 120, pixels are output on clock pulses occurring during horizontal active periods 862). The horizontal front porch period 864 starts after the last active pixel of the line, and lasts until the beginning of the next line (i.e. the beginning of the next horizontal sync). Each of these periods may be defined an integer multiple of the pixel time. Note that the HBI is typically observed for all line times, even those that occur during the VBI. One possible solution for generating the timing for display 160 is to use the clock (i.e., display 120's clock) as the clock during HBIs, which would also allow display 160 to use the input horizontal sync signal and the HBP and HFP periods associated with display 120. As seen in
Turning now to
The YCbCr 4:4:4 pixel data may then be passed to a horizontal chroma downscale unit 552, (e.g. a 2:1 downscaler) which may employ one of a number of possible methods (i.e. sample dropping, simple 2-sample averaging, multi-tap scaling, etc.) to halve the horizontal resolution of the Cr and Cb chroma components. The Y, Cb and Cr components are thus provided to unit 552, which samples the two chroma components at half the sample rate of the luma components, to halve the horizontal chroma resolution). This reduces the bandwidth of the uncompressed signal by one-third, with little to no visual difference perceptible to the human eye. The luma component Y is shown passing through unit 552 to ensure that it experiences the same latency as the chroma components, whatever that latency may be. Thus, unit 552 may produce a luma component output (Y, unchanged and full bandwidth) and a chroma component output (Cb/Cr 524, either Cr or Cb, at half bandwidth). In other words, unit 552 may output a luma value every clock cycle of Clk 512, while outputting either a Cb component or a Cr component during the same clock cycle, alternating between Cr component and Cb component from clock cycle to clock cycle. For example, during a first clock cycle, luma value 522 is output along with a Cr 524 value, during a second clock cycle, luma value 522 is output along with a Cb 524 value, during a third clock cycle, luma value 522 is output along with a Cr 524 value, etc. This is illustrated in
As seen in
Overall, delay elements 560 are responsible for setting the phase offset between the input frame and the output frame.
The pixel processing data path may begin in step 604, in which pixels from an input frame are received in display clock domain (e.g., by converter scaler unit 510). In step 608, if the pixels are received in an RGB color space format, they are converted from the RGB color space into the YCbCr color space. In step 612, the converted pixels are downscaled. More specifically, the chroma components of the YCbCr pixels are downscaled (e.g. halved), while the luma component is passed through unchanged. Then, in step 616, the downscaled converted pixels are output for display in the display clock domain. In one embodiment, a pixel is output when both the output horizontal and vertical active signals are asserted (e.g., during step 428).
The control signal data path begins in step 620, in which one or more control signals in the display clock domain are received. For example, in converter/scaling unit 510, a vertical sync input 564 and a horizontal sync input 566 are received to denote the start of an input frame. In step 624, the received sync signals are delayed, to match a delay resulting from the color space conversion and horizontal chroma downscaling performed by converter/scaling unit 510. The delayed sync signals are output in step 428. The delayed sync signals are output in sync with the downscaled converted pixels output in step 616.
Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
