Not applicable.
The present invention relates generally to an energy efficient display system.
There is a desire among consumers of televisions to watch television content while also being environmentally conscious by reducing the resulting power consumption of the television. In the context of smart grid linked operation, televisions receive signals from a smart meter grid or an energy manager and adjust their operation accordingly. In response to receiving such signals, generally two types of actions are taken. The first action is a time shifting where the television schedules its operation to occur during off peak times. The second action is a demand responsive reduced load operation where the power drawn by the television is reduced by lowering its performance level.
What is desired is a an energy efficient display system while maintaining an image that is readily observable and preferably has pleasing audio.
The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.
In an attempt to make televisions more energy efficient, the principal focus has been on improving device efficiency when in use. Unfortunately, in many cases such improved device efficiency may be insufficient to reach the power reductions desired. Accordingly, in some situations, aggressive power consumption reduction may be desired, e.g., in response to load reduction information from the smart meter. Other aggressive power consumption reductions may be based upon the viewing activity of the viewer. For example, people may only be ‘monitoring’ the television while waiting for something they really want to watch to come on. In some cases people may be in the same room but occupied with another activity, occasionally glancing at the television when some audio suggests something of interests, and they take a glance at the television. In other cases people may walk away from the television and leave television on. Therefore, when such viewer inactivity is detected in front of the television, the television may invoke different power consumption techniques to lower the power consumption, since the viewer will not be as particular about the image quality. In other cases it may be desirable to modify the power used by the audio system.
Under an aggressive power reduction mode, a television is usually dramatically dimmed by reducing the maximum luminance to a lower value. By reducing the maximum luminance, the image presented on the display tends to become very dark with very low contrast (due to black level being held generally constant by the ambient light and display's reflectivity). For backlit liquid crystal televisions (LCD), the luminance reduction is achieved by a reduction in the backlight luminance. For plasma or organic light emitting diode based displays, the luminance is reduced by reducing the power consumed by the active display elements. Other display technologies may reduce power consumption using similar techniques. In any case, the video content and features become less visible to the observer when the display is substantially dimmer when displaying the same image data, making the viewing experience less enjoyable. The principal effect of such dimming is that the contrast of the image is significantly reduced (e.g., normally pegged at black) and many image features that convey important content of the scene falls below or nearly below the visual threshold of the viewer. In the case of mobile devices viewed outdoors with low backlight power, the contrast becomes reduced, but is pegged at white (i.e., the black level rises).
In order to provide a recognizable image while significantly reducing the power consumption, it was determined that rather than attempting to maintain the fine details of the image, it is desirable to modify the image content to be displayed using a non-photorealistic rendering technique. The non-photorealistic rendering technique modifies the image to be more generally cartoon like in its appearance. Such cartoon like images tend to have generally more pronounced edges, regions of more generally uniform color, and/or generally defined by larger region boundaries of the image being defined with edges. The non-photorealistic rendering technique thus may use image processing techniques to identify features of the image in a manner that is constrained by the power usage available. Such cartoon like images may likewise or alternatively include low amplitude details that are rendered as relatively constant, gradual edges that are rendered as steeper edges; and/or darker outlines being rendered along edges.
The power reduction system for a television may include techniques for making the television responsive to a smart meter, connected to smart electricity grid, for providing one or more power savings modes. The power savings mode may include video processing, backlight reduction techniques, and/or power savings by suitable audio processing. In the context of smart grid linked operation, the television receives signals from a smart meter grid, a central server (e.g., any suitable computing device), and/or an energy manager, and adjusts its operation accordingly. In general, two types of actions are taken in response to signals from the smart meter. The first type of action is a time shifting of its operations so that activities occur during off peak times. The second type of action is to reduce the power drawn by the appliance by lowering its performance level.
For a particular implementation of a non-photorealistic rendering technique for video the desirable features to be rendered are preferably selected. In three dimensional shape and depth perception, the desirable features for shape and depth perception include silhouette and contour lines, some contour features such as T-junction and X-junction, ridge and valley lines, and line of curvature. However, the computation of these features require three dimensional data represented in the form of polygons, meshes or three dimensional volume data, and it requires principal directional directions and a surface normal which is computationally intensive to determine. Accordingly, such features are not directly applicable to a system where only two-dimensional data is available and low computational complexity is available.
Rather than relying on three dimensional data, preferably the system incorporates a local-data-driven non-photorealistic rendering technique using two dimensional data, such as television broadcast data. The rendering technique preferably uses less than 10% of full power consumption, more preferably less than 5% of full power consumption, and more preferably less than 2% of full power consumption. The rendering technique preferably extracts and highlights prominent two-dimensional image features to better convey the image content to the observer. The prominent image features may include intensity discontinuity (i.e. edges) and local shape features such as T-junctions and X-junctions. The extracted image features are emphasized in the resulting image. To highlight the essential information, the technique may render prominent image features with the assistance of local contrast stretching. The color of the highlighted edges may be adapted to the local image content.
Referring to
An ambient sensor 110 senses the ambient lighting levels which are received by an ambient analysis module 112. The management module 114 may receive signals from the ambient analysis module 112 to determine, at least in part, sufficient display brightness under low lighting conditions and/or modification of power usage of the television and/or associated devices. This information can be used by the management module 114 to control display brightness, for example, and hence power consumption. An example of power consumption variation with ambient light is illustrated in
The management module 114 may also receive input from a presence analysis 120 which receives input from a presence detector 128 to determine, at least in part, sufficient display brightness and/or modifications to power usage of the television and/or associated devices. Based on the multiple inputs from 106, 112, and/or 120, the television 100 selects actions in response for a global power control 122, video rendering 124, and audio volume control 126. Tables 1 and 2 summarizes one set of input and output options for the management module 114.
The management module 114 may select average power targets and/or rendering mode based upon the power usage desired. The management module 114 may likewise provide data indicative of, in general, power usage to the global power control 122, audio usage to the audio volume control 126, and/or video rendering to the video rendering module 124.
An exemplary global power control 122 is shown in
Referring to
Another technique for reducing the power consumption is to consider peripheral components frequently attached to, and in some cases controlled by, the television. One type of such component is the audio system associated with the television 100. The power consumption of audio with surround sound when using an audio-visual receiver (audio amplifier) can be significant.
power consumption with external AVR can be significant;
power consumption varies within an audio program content;
mean power consumed can be controlled by the volume level setting.
An audio power control technique may be used to control the audio dynamic range to reduce overall power usage. One aspect that the system may control is the volume control, such as setting a volume level for each of the channels. As can be observed from
First, in one embodiment the audio calibration may be done on a set of training audio sample data and an average power consumption may be noted at different volume level settings. Then during the playback phase, the volume level setting may be automatically adjusted to a desired level based on the training phase measurements.
Second, in another embodiment the power consumed for an audio input with constant audio code values may be determined. This may be repeated for different constant audio code input values. Then while playing audio content, an analysis of its audio code values provides an estimate of power being consumed. A modified volume level settings may be selected based on the desired target power consumption.
Referring to
In an audio system the audio content may have X different input audio channels. A down mixing component may take as input X input audio channels and may output Y output audio channels, with Y≦X.
In one embodiment, the number of output channels Y may be selected based on the target power consumption desired. In another embodiment, the down-mixing operation may drop one or more input audio channels to arrive at the target number of Y output audio channels. In another embodiment, a down-mixing operation may mix two or more input audio channels to one audio output channels to arrive at Y audio output channels. Referring again to
The audio volume control 126 may analyze the audio volume levels of each input audio channel (e.g., input surround channels). The audio volume control 126 may compute and/or select shift curves to emphasis and de-emphasis curves {C1, C2, . . . , CX, CX+1} to apply on individual input audio channels. The computation may be performed using information from volume analysis module and/or dynamic range compensation module. Audio channel level shift operation with down-mixing may use information from audio channels level shift curves applied on each input audio channels to generate output audio channels. Thus AjO=Cj(AjI) ∀j, where AjO, AjI respectively denote the audio output and audio input channel j and Cj is the audio channel level shift curve for channel j.
Referring to
Referring to
The left path boosts the brightness of the input color image with an image-content-adaptive, ambient-aware and power-aware brightness boosting technique. The inputs to the brightness boosting path include the original input image, the ambient level given by the ambient sensor (110 in
The input image resolution may be quite high (e.g., larger than full HD resolution), therefore the system preferably low-pass filters the image and down-samples it to a lower resolution 300 to facilitate near real-time processing with limited computation resources. In addition to save computational resources, the low-pass filtering and down-sampling also has the benefit of suppressing noise in the input image, which, if otherwise unprocessed, may react to subsequent processing. An alternative for removing noise is to decompose the image signal into two channels with nonlinear sieve filter or bilateral filter.
In a second step, the system detects edges/contours using gradient estimation 310. The first order gradient can be extracted with various types of gradient operators including Canny, Sobel, Prewitt, and Roberts. In order to extract true contours with large gradients rather than noisy segments, the system may use a large spatial support when computing the gradient at each pixel. For example, the gradient at point p can be set to the largest gradient with a local search in the left, right, top and bottom direction. Depending on the effectiveness of the first order gradient, discontinuities of the first order gradient can also be extracted with a Laplacian operator.
In a third step, the system may analyze the data 320 activity of a local neighborhood to determine how to render the detected edges/gradients: for busy neighborhood 330 where the average of gradients is larger than a threshold, T, the edge will be rendered with its width proportional to its gradient; for flat area 340 where the average of gradients is smaller than T, the detected edges will be removed and white background will be rendered. The threshold T is defined to be the average of the gradients in the entire image.
The system may enhance 350 the visual effects by smoothing the rendered gradients so that the contours are blended with the background and the broken edges are linked. Other enhancement technique may also be used such as local contrast stretching. The system may up-sample the edge map 360 back into the original resolution.
The output from these two paths, one is the gradient and the other is the brightened input, are blended by a linear weighted average. The blending coefficients α is either determined by an automatic a selection algorithm which depends on input content, ambient level and power usage factor, or is selected by the user. The final NPR result is obtained by mapping the code values of the blended image into the range of [0 255].
Another technique receives the input image, first modifies it to an NPR image having the full (or substantially full) range of system code values (e.g., 0-255 for an 8 bit system), and then sends it to the driving stages 400 of the LCD. A separate control signal (dependent on the presence detector's result for viewer's state) goes to the backlight, and indicates whether it should be reduced. This dims the image date on the LCD, but can save substantial power. For Plasma, OLED, or other self-emitting displays, the NPR image may be rescaled to lower amplitude values (from 0-255 to 0-40, for example). The lower max code values results in a dimmer luminance on the display, but with an advantageous power savings reduction.
In addition to the previously described main modes of utilizing an NPR image, a pair of exemplary techniques for generating the NPR image (whether for the 0-255 range or for a pre-dimmed range) are illustrated.
Referring to
The primary steps to generate the cut-out image include first applying a nonlinear tone scale to strongly boost the images brightness. Then a LPF may be applied to smooth the image and reduce low amplitude textures. The filter may end up being quite large. If the input image is already strongly filtered and down sampled then this step may be omitted. Next the number of effective gray levels are reduced to change the image to essential shapes, so that it looks like a multiple color paper cut-out version of the image. One technique is to divide the image by N (typically 64, for an image with range 0-255), quantize (e.g., round to a nearest integer) and then rescale back to the original range (this example will give an image with 4 cutout gray levels per color). The rendered edge map may be performed by the following operations (1) low-pass filtering and downsampling; (2) gradient estimation; (3) local data analysis; and (4) render edge with width proportional to gradient.
The cut-out image is added to the rendered edge image, where the sign of the edge image per pixel is dependent on the per pixel gray level of the cut-out image. That decision is set by the parameter MID, which may be 128 (out of a range of 0-255). The preferred value depends on the display's tone scale. The result will be a non-photorealistic rendered image where there will be dark lines over bright regions, and bright lines over dark regions. This will increase the visibility of the lines and regions, when viewed on a low contrast display (it is low contrast because the power is strongly reduced, making the white max level lower, and hence closer to the black level).
Referring to
Referring to
Referring to
Another type of sensor is an infra-red or visible light sensitive camera with a higher pixel resolution (e.g., 640×480 pixels) than the IR motion sensor. The camera takes periodic snapshots of the environment and can be used to detect human faces in front of the display. The face detection rate is higher when the viewer is looking at the camera directly and lower when the viewer turns his/her head away. This difference in the detection rate can be used to determine if the viewer is looking at the TV directly.
One embodiment is to use an infra-red motion sensor to sense the viewers' motion. However, the single IR motion sensor is sensitive to all kinds of motion and cannot differentiate the different presence modes. Another embodiment is to use an infra-red or visible light camera to sense the viewing environment in front of the TV. The camera can be used to recognize different viewer presence modes.
The preferred embodiment is to combine the infra-red motion sensor and the infra-red camera to sense both the viewer's motion and the viewing environment. Both sensors may be installed on the bezels of the TV or inside the TV pixel array. The motion sensor is turned on all the time and reports whether there is motion in the past second. The camera is turned on at a user-specified interval ΔT (e.g., 30 seconds) and captures a snapshot of the viewing environment. The face detection technique may be applied to find the faces in the image.
A face occurrence frequency is computed for a period of time:
The face occurrence frequency is a number between 0% and 100%. Two thresholds, Fhigh and Flow, may be used to decide the viewer presence mode based on the frequency. If there is a viewer constantly watching the TV, the frequency will be higher than Fhigh; otherwise if the viewer is away, the frequency will be close to zero and lower than Flow. For the other two viewer presence modes, the frequency will lie between Fhigh and Flow. Typical numbers of Fhigh and Flow can be 80% and 20%. These numbers can be adjusted by the viewers.
Based on the combined sensors an energy management scheme is provided. First, the technique checks 400 if there is any recent viewer controlling action in a past period of time T (e.g., 30 seconds). If the viewer makes any controlling action (e.g., pressing any buttons on the TV remote), it is assumed that the viewer is paying attention to the TV and the presence mode is “watching” 402. Otherwise, the face occurrence frequency 404 is computed and used to make the decision. If the face occurrence frequency 406 is higher than Fhigh, the viewer is looking directly at the TV more often, therefore in the “watching” mode 402.
Otherwise the face occurrence frequency 408 is compared to Flow. If the viewer is looking at TV intermittently (frequency>Flow) 408, he/she may be doing some other activities; the mode is “peeking” 410. If the viewer seldom looks at TV (frequency>Flow), the IR motion sensor is used to decide the mode. If there is any motion sensed 412 by the IR motion sensor, the viewers may be moving in the viewing environment and want to hear the sound from the TV; the mode is “listening” 414. Otherwise, if there is no detected face and no motion in the scene, the viewers are probably not in the environment and the mode is “away” 416. In some cases, the system will attenuate lower frequency, values to reduce power consumption.
When the viewer presence mode changes, the corresponding energy management scheme is also changed. For the “watching” mode 402, viewers want to have the full viewing experience and therefore both image and sound are generated at 100%. When the viewers are only “peeking” 410, the images can be rendered at an energy saving mode and the sound is still output at 100%. For the viewers who are only “listening” 414, the images are turned off on the TV while the sound is still generated at 100%. If the viewers are “away” 416, the image is turned off to save energy while the sound level can be reduced or even set at 0%, depending on the viewer's presence. The audio control module uses the input from the viewer presence module to make further decisions.
As a general matter, the system may include a pair of image rendering techniques. One of the techniques may be the non-photorealistic rendering technique as previously described. Another of the techniques may be generally known brightness preservation techniques. The brightness preservation techniques tends to attenuate higher luminance values in a manner while not similarly attenuating lower luminance values. The curves for the brightness preservation techniques tend to be similar in appearance to
The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.