Field of the Invention
This invention relates generally to digital video displays, and more particularly to displays with features that improve latency.
Description of the Background Art
Liquid crystal displays typically include large arrays of individual pixels. Intensity values to be displayed by each pixel are typically represented by multi-bit data words, and each bit of the multi-bit data word is asserted on the pixel during a portion of a video frame time corresponding to the significance of the asserted bit. Each bit will cause the pixel to display either a light (“on”) or dark (“off”) intensity, depending on the values of the asserted bits. Over the frame time, an observer's eye integrates the light and dark intensities of the individual bits, to perceive an intermediate intensity value corresponding to the value of the multi-bit data word.
The process of loading each data bit onto each pixel takes some time. The latency of a display is defined as the amount of time between the receipt of a first portion of a frame of image data into a buffer of the display and the first assertion of data of that frame on the pixels of the display.
The increase in interactive display technology, such as computer monitors, video game consoles, and virtual reality headsets, among others, has created a need for systems with decreased latency. In such technology, video data must be altered as the user interacts with the device or with the world. For example, a head mounted display might display information over an object in the user's field of view. If the information is intended to be displayed in a fixed position with respect to the world, image data from the device must be constantly altered as the user moves his/her head or eyes with respect to the object. In known devices, visual artifacts (blurred or jerky object motion) are caused by the latency of the display device.
Efforts to improve latency in liquid crystal displays have not been fully satisfactory. While some have decreased latency in certain display devices, the delays remain significant in applications requiring immediate alteration of image data. For example, in known devices there is at least a one frame delay between receiving the image data and displaying it. What is needed, therefore, is a system and method for decreasing latency of a display to less than the frame time of the display.
The present invention overcomes the problems associated with the prior art by providing a digital display and a novel method of writing data to the display. The invention facilitates a significant reduction in the latency of the display, by reducing the amount of data that is loaded into a display driver before a given frame is displayed. In one embodiment of the present invention, the size of frames of image data is reduced to generate frames of down-scaled image data, which contain many fewer data words than the original image data. The down-scaled image data can be loaded into the driver, and consequently displayed by the display, much quicker than the original image data.
In another embodiment, portions of frames of image data are displayed before the entire frame of image data is loaded into the display driver. For example, less than a full red color field of an initial frame is displayed after only a portion of the full initial frame of data is received. Then, less than a full green color field of the initial frame is displayed before the full initial frame of data is received. Finally, a full blue color field is displayed. The remaining two-thirds of red data and one-third of green data of the initial frame is displayed with the next frame of data, portions of which are also displayed before the entire next frame is loaded into the display driver.
An example display system includes a display, including a plurality of individual pixels, and a display driver. The display driver is coupled to receive sequential frames of image data and to assert at least a portion of the image data on the pixels of the display. The display driver also asserts at least some data corresponding to each frame of the image data on the pixels of the display within a predetermined amount of time after receiving a first portion of each frame of the image data. The predetermined amount of time is less than an amount of time required for the display driver to receive a full frame of the image data.
In an example embodiment the display system includes an image data scaler, which is electrically coupled to receive the frames of image data and operative to reduce the size of the image data. Each frame includes a particular amount of data, and in reducing the size of the image data, the image data scaler generates down-scaled frames of image data, which include less than the particular amount of data. The image data scaler also provides the down scaled frames of image data to the display driver. The display driver is electrically coupled to receive the down-scaled frames of image data from the image data scalar, and is operative to increase the size of the down-scaled frames of image data. In increasing the size of the down-scaled frames of image data, the display driver generates up-scaled frames of image data and also asserts at least some data from each of the up-scaled frames of image data on pixels of the display before receiving an amount of data corresponding to the particular amount of data.
In a particular embodiment, the down-scaled image data includes no more than 75% of the amount of image data in a frame of the originally formatted image data. In another more particular embodiment, the down-scaled image data includes no more than 50% of the amount of image data in a frame of the originally formatted image data.
In another more particular embodiment, the image data scaler is operative to generate the down-scaled data by omitting data values associated with particular pixels of the display from the originally formatted data. In yet a more particular embodiment, the image data scaler is operative to generate the down-scaled data by omitting data values associated with predefined rows of pixels of the display from the originally formatted data.
Optionally, the display system includes a controller operative to dynamically transition the image data scaler between an on-state and an off-state. For example, one embodiment includes a sensor. The controller, responsive to data from the sensor, transitions the image data scaler between the on-state and the off-state. In a more particular embodiment, the sensor is an image sensor. In another more particular embodiment, the sensor is a motion sensor. In yet another more particular embodiment, the sensor is an orientation sensor.
In another example embodiment, the display system includes an image data buffer electrically coupled to receive the sequential frames of image data. The image data buffer provides the sequential frames of image data to the display. The display driver is operative to assert a portion of a first frame of image data on a first set of the pixels of the display, and assert a portion of a second frame of image data on a second set of pixels. The portion of the second frame is asserted on the second set of pixels while the portion of the first frame is being asserted on the first set of pixels. The display is illuminated with light of a first color while the portion of the first frame and the portion of the second frame are being simultaneously asserted on the pixels of the display. The display is illuminated before the entire second frame is received in the image data buffer. In a particular embodiment, the portion of the first frame is larger than the portion of the second frame.
In another particular embodiment, the display driver asserts a second portion of the first frame on a third set of pixels, and also asserts a second portion of the second frame on a fourth set of pixels. The second portion of the second frame is asserted on the fourth set of pixels while the second portion of the first frame is being asserted on the third set of pixels. The display is illuminated with light of a second color while the second portion of the first frame and the second portion of the second frame are being simultaneously asserted on the pixels. In a more particular embodiment, the second portion of the first frame is smaller than the second portion of the second frame.
In yet another particular embodiment, the display driver asserts a third portion of the second frame on all of the pixels. The display is illuminated with light of a third color while the third portion of the second frame is asserted on all of the pixels.
In another particular embodiment, one of the first set of pixels and the second set of pixels includes one-third of the pixels of the display. The other of the first set of pixels and the second set of pixels includes two-thirds of the pixels of the display. In yet another particular embodiment, one of the first set of pixels and the second set of pixels includes a middle one-third of the pixels of the display. In another particular embodiment, one of the first set of pixels and the second set of pixels includes one-fourth of the pixels of the display, and the other of the first set of pixels and the second set of pixels includes three-fourths of the pixels of the display.
A method for displaying digital data is also described. The method includes receiving sequential frames of image data, and asserting at least some data corresponding to each frame of the image data on pixels of a display within a predetermined amount of time after receiving a first portion of each frame of the image data. The predetermined amount of time is less than an amount of time required to receive each full frame.
An example method includes receiving frames of originally formatted image data, reducing the size of the image data to generate frames of down-scaled image data, providing the frames of down-scaled image data to a display, increasing the size of the frames of down-scaled image data to generate frames of up-scaled image data, and asserting at least some data from each of the frames of up-scaled image data on pixels of the display. The originally formatted frames of image data each include a particular amount of data. The down-scaled frames of image data each include less than the particular amount of data. The data is asserted on the pixels of the display before an amount of data corresponding to the particular amount of data is received.
A particular method includes asserting at least some data of each frame of the up-scaled image data on the pixels of the display within a certain amount of time after receiving a first portion of each frame of the down-scaled image data. The certain amount of time is less than an amount of time required for the display to receive a frame of the originally formatted data.
In another particular method, the step of reducing the size of the image data to generate frames of down-scaled image data includes generating frames of down-scaled image data each including no more than 75% of the amount of image data in a frame of the originally formatted image data. In a more particular embodiment, the step of reducing the size of the image data to generate frames of down-scaled image data includes generating frames of down-scaled image data including no more than 50% of the amount of image data in a frame of the originally formatted image data.
In yet another particular method, the step of reducing the size of the image data to generate frames of down-scaled image data includes omitting data values associated with particular pixels of the display from the originally formatted data. In another particular method, the step of reducing the size of the image data to generate frames of down-scaled image data includes omitting data values associated with predefined rows of pixels of the display from the originally formatted data.
In yet another particular method, the step of reducing the size of the image data includes reducing the size of the image data when a predefined condition is met and not reducing the size of the image data when the predefined condition is not met. Particularly, the step of reducing the size of the image data includes receiving data from a sensor and determining whether the condition is met based at least in part on the data from the sensor. In an even more particular method, the step of receiving data from a sensor includes receiving data from an image sensor. In another more particular method, the step of receiving data from a sensor includes receiving data from a motion sensor. In yet another more particular method, the step of receiving data from a sensor includes receiving data from an orientation sensor.
Another example method includes the steps of asserting a portion of a first frame of image data on a first set of the pixels of the display and asserting a portion of a second frame of image data on a second set of pixels. The portion of the second frame is asserted on the second set of pixels while the portion of the first frame is being asserted on the first set of pixels. The method also includes illuminating the display with light of a first color while the portion of the first frame and the portion of the second frame are being simultaneously asserted on the pixels. The display is illuminated before the entire second frame is received in the image data buffer. In a particular method, the portion of the first frame is larger than the portion of the second frame.
Another particular method includes the steps of asserting a second portion of the first frame on a third set of pixels and asserting a second portion of the second frame on a fourth set of pixels. The second portion of the second frame is asserted on the fourth set of pixels while the second portion of the first frame is being asserted on the third set of pixels. The method also includes illuminating the display with light of a second color while the second portion of the first frame and the second portion of the second frame are being simultaneously asserted on the pixels of the display. In a more particular method the second portion of the first frame is smaller than the second portion of the second frame.
Yet another particular method includes the steps of asserting a third portion of the first frame on a fifth set of pixels and asserting a third portion of the second frame on a sixth set of pixels. The third portion of the second frame is asserted while the third portion of the first frame is being asserted on the fifth set of pixels. The method also includes illuminating the display with light of a third color while the third portion of the first frame and the third portion of the second frame are being simultaneously asserted on the pixels of the display.
In an even more particular method, the third portion of the first frame includes 0% of the first frame and the third portion of the second frame includes 100% of the second frame. Additionally, the fifth set of pixels includes none of the pixels of the display and the sixth set of pixels includes all of the pixels of the display.
In another particular method, one of the first set of pixels and the second set of pixels includes one-third of the pixels of the display. The other of the first set of pixels and the second set of pixels includes two-thirds of the pixels. In yet another particular method, one of the first set of pixels and the second set of pixels includes a middle one-third of the pixels. In another particular method, one of the first set of pixels and the second set of pixels includes one-fourth of the pixels, and the other of the first set of pixels and the second set of pixels includes three-fourths of the pixels.
Another example method for displaying digital video data is also described. The method includes loading sequential frames of video data into a data buffer. Each frame of video data includes a plurality of color fields. The method also includes asserting a first portion of a first color field of a first frame of the video data onto a display and illuminating the display with a first colored light corresponding to the first color field. The first portion of the first color field is asserted after the first portion of the first color field is loaded into the data buffer and before an entirety of the first frame is loaded. The method also includes asserting a first portion of a second color field onto the display and illuminating the display with a second colored light corresponding to the second color field. The first portion of the second color field is asserted after the first portion of the second color field is loaded into the data buffer and before the entirety of the first frame is loaded. The method further includes asserting a first portion of a third color field onto the display and illuminating the display with a third colored light, corresponding to the third color field. The first portion of the third color field is asserted after the first portion of the third color field is loaded into the data buffer.
Additionally, the method includes asserting a first portion of the first color field of a second frame of video data and a remaining portion of the first color field of the first frame onto the display and illuminating the display with the first colored light. The first portion and the remaining portion are asserted after the first portion of the first color field of the second frame is loaded into the data buffer and before an entirety of the second frame is loaded. The method further includes asserted a first portion of the second color field of the second frame and a remaining portion of the second color field of the first frame onto the display and illuminating the display with the second colored light. The first portion and the remaining portion are asserted after the first portion of the second color field of the second frame of video data is loaded into the data buffer and before the entirety of the second frame is loaded. Additionally, the method includes asserting a first portion of a third color field of the second frame and any remaining portion of the third color field of the first frame into the display and illuminating the display with the third colored light. The first portion and the remaining portion are asserted after the first portion of the third color field of the second frame is loaded into the data buffer.
The present invention is described with reference to the following drawings, wherein like reference numbers denote substantially similar elements:
The present invention overcomes problems associated with the prior art, by providing a system and method for decreasing display latency that includes manipulating image data and the way it is written to the display. In the following description, numerous specific details are set forth (e.g., the specific order in which portions of data are written, the structure of the data buffer, etc.) in order to provide a thorough understanding of the invention. Those skilled in the art will recognize, however, that the invention may be practiced apart from these specific details. In other instances, details of well-known digital image display practices (e.g., planarization, routine optimization, etc.) and components have been omitted, so as not to unnecessarily obscure the present invention.
In alternate embodiments, host 102 can be any of a number of various devices. For example, host 102 can be a cell phone, a head mounted display, or any other similar device. Additionally, host 102 can have display 104, data source 106, and sensor 108 embodied therein. Consequently, host 102 can include any number of devices (e.g. cameras, microphones, motion/orientation sensors, etc.), whether or not they are explicitly stated. Additionally, host 102 can serve any number of functions concurrently with the display functions described in this disclosure.
In addition, system 100 can include any combination of various devices. For example, in alternate embodiments, system 100 can include a cell phone (host 102), with data source 106 embodied therein, and a head mounted display with display 104 and sensor 108 embodied therein. In such embodiments control bus 112 can be replaced with any applicable datalink, including, but not limited to, a short range wireless (e.g., Bluetooth®) connection, a wired connection (e.g., a universal serial bus (USB)), and so on.
Video controller 202 receives data from sensor 108 and uses that data to control the function of data scaler 200, via a control bus 204. For example, as a user moves his/her head (or eye direction) relative to his/her surroundings, video controller 202 sends control signals to data scaler 200 to initiate the down-scaling of image data in order to improve display latency. On the other hand, when a user's head or eye direction remains fixed relative to his/her surroundings, video controller 202 sends control signals to data scaler 200 to suspend the down-scaling of image data in order to improve image resolution. Additionally, video controller 204 sends control signals to display 104 via a control bus 112. Control signals sent to display 104 include signals to coordinate video data transfer, and a signal to indicate the state of data scaler 200.
Controller 300 also coordinates the transfer of data from data load register 302 into data buffer 304, via a data bus 314. When a full row of data is loaded into data load register 302, controller 300 asserts control signals (e.g., a row output signal) on control bus 312 and control signals (e.g., a row address and a row enable signal) on control bus 316. These control signals cause a row of data to be asserted by data load register 302 onto data bus 314 and also cause data buffer 304 to latch the asserted row of data into data buffer 304.
When data scaler 200 is functioning, each row of down-scaled video data is latched into data buffer 304 twice, resulting in up-scaled data being written to data buffer 304. For example, if the down-scaled video data includes only the odd rows of video data, then each odd row of video data will be written into its appropriate position in data buffer 304 and also into an adjacent position for an even numbered row of video data. In other words, each odd numbered row of video data will be substituted for an adjacent even numbered row of video data. Controller 300 generates the up-scaled frame of video data by duplicating the rows of data in the down-scaled image data.
By latching each row of data into data buffer 304 twice, the amount of time required to load an entire frame of data is reduced by approximately one-half. Because controller 300 can begin asserting the data in data buffer 304 on pixel array 306 as soon as the frame of up-scaled video data is latched into data buffer 304, the latency period is half of the full latency period, and a half-frame time decrease in latency is sufficient to prevent visual artifacts in the displayed images.
Data buffer 304 contains enough storage space to hold two frames of image data. This space is partitioned into two halves, which are alternately written to and read from (double buffer scheme). After a frame of image data is written to data buffer 304, that data is written to pixel array 306, via a data bus 318, while a subsequent frame is being written to data buffer 304, via data bus 314. Once the first frame is displayed, the subsequent frame of data is then written to pixel array 306, via data bus 318, while yet another frame is being written to data buffer 304. This process is repeated for as long as image data is received from host 102.
Based on the image data asserted from data buffer 304, pixel array 306 modulates and reflects light from light source 308 to a display screen (not shown), through optics (not shown), or directly into a user's eye. Light source 308 is an RGB light source operative to sequentially emit red, green or blue light onto pixel array 306, in order to create colored images. Light source 308 can include, for example, light emitting diodes, lasers, or any other suitable colored light sources.
Controller 300 provides control signals to light source 308, via a control bus 322, in order to coordinate the function of light source 308 and pixel array 306. Colored images are generated by displaying separate images in three different colors in quick succession. The human eye blends the three colors, and the perceived color of any given pixel is a function of the intensity of that pixel in each of the three colors. Controller 300 coordinates the flashing on of each colored light of light source 308 with the assertion of corresponding data onto pixel array 306.
Red, green and blue image data of a particular frame is written to portions 400(1), 402(1), and 404(1), or to portions 400(1), 402(2), and 404(2). Typically, image data is received in 24-bit data words, each corresponding to a particular pixel. Every 24-bit data word is separated into 8 red bits, 8 green bits, and 8 blue bits, which are written to portions 400(1) or 400(2), 402(1) or 402(2), and 404(1) or 404(2), respectively. When data scaler 200 is functioning, some incoming data bits will be written more than once. Which data bits are duplicated depends on which data bits were originally omitted in down-scaling the image data. For example, if the originally formatted data was down-scaled by omitting data corresponding to every other row of pixels, each row of the down-scaled data would be duplicated as it is written into image data buffer 304. This method allows down-scaled data to be written to image data buffer 304 in half the time as compared to the originally formatted data. Each frame of this new, up-scaled data fills one of halves 406(1) or 406(2).
The data stored in each portion of buffer 304 corresponds to a single color field, associated with one of the three colors that make up the frame. Because the three color fields that make up a single frame are displayed one at a time, data is written to pixel array 306 from only one of the portions at a time. Data is written from one of halves 406(1) or 406(2), while data is written to the other of halves 406(1) or 406(2). By writing frames to alternating halves of data buffer 304, unnecessary delays are avoided, and latency is minimized.
Data buffer 706 contains enough storage space to hold two frames of image data. This space is partitioned into two halves and is used as a dual buffer. However, controller 702 can transfer data into one portion of a particular half of data buffer 706, while transferring data out of another portion of the same half of data buffer 706. After controller 702 writes a first portion of an initial frame of image data into data buffer 706, controller 702 asserts control signals (e.g., a row address and a row output signal) on control bus 718 and control signals (e.g., a row address and a row enable signal) on control bus 722, to transfer that data to pixel array 708 via a data bus 720, while the remaining portion of the initial frame continues to be transferred into data buffer 706 via data bus 716.
Only a portion of the initial frame is displayed in some of the color fields, and as more and more of the initial frame is written to data buffer 706, the portion of the initial frame data displayed in each color field increases. While the third color field of the first frame is displayed, a first portion of a subsequent frame is written to data buffer 706. The first portion of the subsequent frame is then displayed along with the previously un-displayed portion of the first color field of the initial frame. As more and more of the subsequent frame is written to data buffer 706, the portion of the initial frame displayed in each color field decreases and the portion of the subsequent frame displayed increases. This process is repeated for as long as image data is received from host 600.
Because the time between frames is small (a display working at 60 frames-per-second must display a frame every 60th of a second), the human eye cannot detect that two portions of separate images are being displayed at once. The frames blend together smoothly. Optionally, the initial portion of image data that is displayed can consist of the middle of the frame. Because the eye is generally focused near the middle of the screen, this alternative further increases the perceived smoothness of the video and reduces the perception of any field tearing. This method will be explained in more detail as part of the description of
As briefly described above, controller 702 also coordinates the transfer of data from data buffer 706 to pixel array 708. For example, controller 702 asserts control signals on control bus 718, to cause data buffer 706 to assert a row of image data onto data bus 720. Controller 702 also asserts control signals onto control bus 722, causing pixel array 708 to latch the row of image data being asserted on data bus 720 by data buffer 706. In general, controller 702 asserts row addresses, row enable signals, and any other necessary control signals onto control buses 718 and 722 to cause the image data to be asserted on the appropriate pixels of pixel array 708 in the order and sequences described herein.
Based on the image data received from data buffer 706, pixel array 708 modulates and reflects light from light source 710 to a display screen (not shown), through optics (not shown), or directly into a user's eye. Light source 710 is an RGB light source operative to selectively emit red, green and blue light onto pixel array 708, in order to create a series of colored images.
Controller 702 provides control signals to light source 710, via a control bus 724, in order to coordinate the function of light source 710 and pixel array 708. Colored images are generated by displaying separate images in three different colors in quick succession. The human eye blends the three colors, and the perceived color of any given pixel is a function of the intensity of that pixel in each of the three colors. Controller 702 coordinates the emission of each colored light of light source 710 with the assertion of corresponding data on pixel array 708.
Those skilled in the art will recognize that the division of the data into thirds is not an essential element of the present invention. For example, the present invention may be practiced by displaying frames in quarters instead of thirds. Instead of displaying the blue portions of frame data in individual blue color fields, the blue portions would be split into three quarters of new frame data and one quarter of old frame data. Likewise, the green fields would be split in half, and the red fields would consist of one quarter of new frame data and three quarters of old frame data. This implementation would further decrease the latency, and is explained in more detail with reference to
A first green output field 904(1) and a first blue output field 906(1) are displayed immediately following the display of output field 902(1). Output field 902(1) includes a red portion 908(1) and a prior data portion 910(1). Prior data portions include data from a prior frame or, in the case of an initial frame, random data from the data buffer 706 that is not displayed. Similarly, output field 904(1) includes a green portion 912(1) and a prior data portion 914(1), and output field 906(1) includes a blue portion 914(1). A second red output field 902(2) is displayed following the display of output field 906(1). Output field 902(2) includes a red portion 908(2) of data from frame 900(2), and a red portion 910(2) of data from frame 900(1). A second green output field 904(2), including a green portion 908(2) and a green portion 910(2), is displayed following the display of output field 902(2). Green portion 908(2) includes data from frame 900(2) and green portion 910(2) includes data from frame 900(1). A second blue output field 906(2) is displayed following the display of output field 904(2). Output field 906(2) includes data from frame 900(2). The process of displaying portions of frames as they are written into data buffer 706 continues until there is no more video data to display.
Output fields 902(2), 904(2), and 906(2) are displayed in succession following the display of output field 906(1) and a blank (off) time. Output field 902(2) includes a red portion 918(2), which includes image data from a subsequent frame of data, and a red portion 920(2), which replaces prior data portion 920(1) and includes previously un-displayed image data from the previous frame. Output field 904(2) includes a green portion 922(2), which includes image data from the subsequent frame, and a green portion 924(2), which replaces prior data portion 924(2) and includes image data from the previous frame. Output field 906(2) includes a blue portion 926(2), which is comprised of image data from the subsequent frame, and a blue portion 928(2), which replaces prior data portion 928(1) and is comprised of image data from the previous frame. The process of displaying portions of frames as they are written into data buffer 706 continues until there is no more video data to display.
The use of the fourth output field 930 facilitates options other than the use of another color field. For example, the fourth color field can be used to display data of one (e.g. green) of the original three colors twice for each frame to implement an RGBG scheme. When the same green data is displayed twice in a frame, the intensity of the green light source is dimmed to achieve the proper over all intensity of the green field. As yet another option, the fourth field can facilitate the display of additional numbers of bits for one of the colors, in order to accommodate higher resolution input data.
In the present embodiment, the data received from the sensor makes up at least part of the criteria for determining whether or not the condition of step 1200 is met. Those skilled in the art will recognize, however, that there are a number of possible criteria for making the determination of step 1204. For example, the determination can be made based solely on the image data. Alternatively, the determination can be made based on user input. Ultimately, any criteria that bear upon the quality of the displayed video can be used to make the determination of step 1204.
The description of particular embodiments of the present invention is now complete. Many of the described features may be substituted, altered or omitted without departing from the scope of the invention. For example, alternate data buffers, may be substituted for the data buffers described in
This application claims the benefit of priority to currently pending U.S. Provisional Patent Application No. 62/273,558, filed by the same inventor on Dec. 31, 2015, which is incorporated herein by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62273558 | Dec 2015 | US |