Patent Application
20130278642
The present disclosure relates generally to reduction of energy consumption in wireless mobile communication devices and, more specifically, to content-based display adaptation control for video content displayed on a wireless mobile communication device.
In recent years, display resolution on mobile devices has advanced significantly, to where 720p or even higher super liquid crystal display (LCD) or OLED organic light emitting diode (OLED) displays are or soon will be mainstream for smart phones and tablets. However, such high display resolution requires much more energy for rendering, especially for video where high frequency frame buffering and display panel refresh are indispensable.
For LCD displays, power consumption is a monotonic function of the backlighting brightness level; for OLED displays, power consumption is controlled by the supply voltage as well as the display content itself. While a brightness control is already implemented on some mobile devices, those controls typically must be adjusted prior to issuing a new job—that is, before starting playback of a video. For example, brightness may be set at 100%, 50%, or even 25% prior to watching a video, but cannot be changed dynamically without interrupting playback of the video. In addition, since power consumption is determined by the supply voltage and input image for OLED displays, current implementations do not provide a mechanism for adapting the voltage.
There is, therefore, a need in the art to improve mobile device displays by allowing either LCD display backlighting brightness or OLED supply voltage to be adapted according to the content being displayed, saving significant display energy.
Segments for a video are transmitted in payload units with an extended network abstraction layer unit (NALU) header or supplemental enhancement information (SEI) message within which is embedded display adaptation information that may be employed to control display brightness and thereby reduce power consumption during display of the respective segment. The display adaptation information includes at least a maximum pixel brightness that may be used to scale pixel brightness and correspondingly reduce backlighting for liquid crystal displays, or to adjust the supply voltage for OLED displays. The maximum pixel brightness is set to a level saturating a portion of the pixel histogram without perceptual loss to the viewer, resulting in further reduction of power consumption.
Before undertaking the DETAILED DESCRIPTION below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or,” is inclusive, meaning and/or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, where such a device, system or part may be implemented in hardware that is programmable by firmware or software. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.
For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts:
The metadata used for display adaptation can be embedded into the video stream as the extended NALU (network abstraction layer unit) header.
In the present disclosure, display adaptation is embedded within the video content information using an extended Network Abstraction Layer (NAL) Unit (NALU) header or supplemental enhancement information (SEI) message, which is then parsed at the decoder to help with display power reduction. For LCD displays, the display brightness is adjusted, while for OLED displays, the display supply voltage is adapted. Elements in this extended NALU header or SEI message can be derived at the encoder during video encoding.
Display adaptation is defined by enabling a NALU header or SEI message that can be inserted into stream frame by frame, or group of pictures (GOP) by GOP, scene by scene, or even time interval by time interval, depending on the underlying applications and the hardware capability. By comparison with a frame-level solution, GOP, a scene or time interval based approach requires less overhead for message insertion. For processors that do not support high-frequency display adaptation, e.g., every 33 millisecond (ms) for a 30 Hertz (Hz) video, GOP, scene or time interval based schemes are better than a frame based solution. Nonetheless, the concept is explained herein primarily using a frame level solution.
In H.264 Advanced Video Coding (AVC, also referred to as Motion Picture Experts Group 4 Part 10 or “MPEG-4 Part 10”) and its extensions, each NALU 210, 202 is byte-aligned. The NALU header 202 is either 1 byte or 4 bytes, depending on whether the NALU is a regular single layer packet or a scalable packet. As shown in TABLE I below, the current definition of the NALU header is modified by extension to support embedding of display adaptation related information. TABLE I shows the extended NALU syntax and the corresponding parsing process for H.264/AVC and its extensions (modifications shown in italics in TABLE I):
The tolerable distortion (“distortion percentage”) is used for saturation purposes as described in further detail below.
As illustrated in
H.264/AVC defines various nal_unit_type values for appropriate parsing and decoding, with values 24 through 31 previously left unspecified. A new nal_unit_type=25 is introduced indicating the display adaptation associated information. (The choice of nal_unit_type=25 is merely for the purposes of illustration in this example; any of the “unspecified” nal_unit_type values in TABLE II could be used instead). Once nal_unit_type=25, the command display_adaptation( ) is used to parse and initialize the display adaptation associated data and structure. Each time this nal_unit_type is encountered, the decoder parses the respective NALU header and enables the frame-level, GOP-level, scene-level or time interval-level adaptation. TABLE II shows the extended nal_unit_type definitions in H.264/AVC (modifications shown in italics in TABLE II):
Display
adaptation
non-
non-VCL
Display_adaptation ( )
VCL
Video Coding Layer (VCL) NALUs consist of the video data, slice layer or below; non-VCL information like sequence parameter sets, picture parameter sets, Supplemental Enhancement Information (SEI), etc. may also be provided via a NALU.
Instead of the extended NULA header(s) described above, SEI message(s) may be inserted in the payload bitstream as described in further detail below. A new SEI message with payloadType=47 as shown in TABLE III below. (The choice of payloadType=47 is merely for the purposes of illustration in this example; any previously unspecified payloadType value could be used instead). Each time the SEI message is encountered in the bitstream, the decoder parses that SEI message and enables the frame-level, GOP-level, scene-level or time interval-level display adaptation as defined in TABLE III.
The current definition of the SEI message is modified by extension to support embedding of display adaptation related information. TABLE III shows the extended SEI message for H.264/AVC and its extensions (modifications shown in italics in TABLE III):
TABLE IV shows the display adaptation SEI message syntax in H.264/AVC (modifications shown in italics in TABLE IV):
display_adaptation( payloadSize ) {
}
As evident from TABLES I through IV, three different types of display adaptation (“display scaling method”) are contemplated: display adaptation preserving brightness of the pixels (“BRIGHTNESS_PRESERVED”); display adaptation preserving contrast (“CONTRAST_PRESERVED”); and perceptually lossless display adaptation (“PERCEPTUAL_LOSSLESS”). Display adaptation preserving brightness takes a single value as a parameter: the maximum pixel brightness value (“max_pixel_value”) within a histogram of pixel brightness values for a reconstructed frame encoded with the respective NALU header. Display adaptation preserving contrast rightness takes as parameter both the maximum pixel brightness value and the minimum pixel brightness value (“min_pixel_value”) within the histogram of pixel brightness values for the reconstructed frame. Perceptually lossless display adaptation, preserving both brightness and contrast, takes three parameters: the maximum and minimum pixel brightness values (“max_pixel_value”) within the histogram and the step size (“pixel_hist_stepsize”) of pixel brightness values used in generating the histogram.
In ITU VCEG and International Standards Organization (ISO)/International Electro-technical Commission (IEC) Motion Pictures Expert Group (MPEG) Joint Collaborative Team on Video Coding (JCT-VC) standard H.265 High Efficiency Video Coding (HVEC), the byte stream framework remains the same (i.e., NAL units are employed), but the NAL unit header is longer (and not compatible to H.264), new NAL unit types are introduced and several type number changes are made, and a modified NALU payload syntax is employed (that is also not H.264-compliant). However, both SEI and video usability information (VUI) metadata are permitted. Accordingly, notwithstanding the differences, those skilled in the art will understand how the above-described techniques may be readily adapted for use with HVEC streams.
where BIT_DEPTH is the video bit depth, normally BIT_DEPTH=8. Meanwhile, a lower brightness backlight or supply voltage at bnew=b/η or Vnew=V/η may be used for a net reduction in energy.
Before decoding every frame, the extended NALU header or SEI message is parsed to extract the maximum pixel value used to scale up a current reconstructed frame by (255/max_pixel_value). Let p(i) indicate the original brightness of an i-th pixel value (in raster scan order) in a histogram of pixel brightness for a reconstructed frame as illustrated in
pnew(i)=p(i)*Y, (2)
where Y=(255/max_pixel_value) and max_pixel_value is the parameter specified in the extended NALU header as described above. As apparent by comparison on
Meanwhile, by increasing the pixel brightness, a lower brightness backlighting (for LCD displays) or a lower supply voltage (for OLED displays) may be used for a net reduction in energy. That is, for LCD displays the scaled pixel brightness is employed together with a reduced backlighting brightness. The scaled value may be set at the ratio (max_pixel_value/255)*100%. That is, the scaled backlighting brightness bnew is:
bnew=b/Y, (3)
where b is the original backlighting brightness, and the scaled supply voltage is:
Vnew=V/Y, (4)
where V is the original supply voltage. While linear scaling of backlight brightness and supply voltage are assumed above, in actual implementations the scaling could be non-linear. Either linear or non-linear adjustment may be implemented through a look-up table, which may be constructed by measuring the display power at different levels of the backlight brightness or supply voltage.
To further reduce energy, the maximum pixel value may be further altered to allow some pixel distortion (i.e., saturated after scaling), but without any perceptual difference to the viewer. Instead of using the original max_pixel_value, the fact that the human perceptual visual system (HVS) can tolerate a certain pixel distortion is exploited. A new max_pixel_value is set, max_pixel_value_new:
max_pixel_value_new=max_pixel_value*λ (5)
with λ indicating the perceptual lossless threshold. As example empirical analysis has demonstrated that λ=0.95 does not introduce any perceptual loss. In another words, 5% of the pixels close to the max_pixel_value are either scaled to 255 (the maximum possible pixel brightness) or saturated for 8-bit video.
The parameter min_pixel_value may be similarly employed, together with max_pixel_value, for adaptation when scaling in CONTRAST_PRESERVED mode. The range between maximum and minimum pixel brightness may both be adjusted to maintain contrast. Likewise, the parameters min_pixel_value and pixel_hist_stepsize, together with max_pixel_value, for adaptation when scaling in CONTRAST_PRESERVED mode. The range between maximum and minimum pixel brightness and the distribution of pixel brightness may all be adjusted.
For LCD displays with separate backlighting of each of the red (R), green (G) and blue (B) color channels, pixel brightness scaling and backlighting brightness reduction as described above may be implemented separately for the pixel and backlighting brightness of each of the RGB colors individually. To the extent that separate supply voltages are employed for red, green and blue LEDs within an OLED display, pixel brightness scaling and supply voltage reduction as described above may be implemented separately for each RGB color. In that manner, different color components may be individually adapted.
The histogram of pixel brightness is determined for the video data of the segment being processed (step 602), including determination of at least max_pixel_value, and optionally also min_pixel_value and pixel_hist_stepsize. An extended NALU header or SEI message is generated for the segment of video data being processed (step 603), with the scaling method and appropriate parameters included. For perceptually lossless, saturated display adaptation, the max_pixel_value is set to saturate a portion of the pixels having a value close to the actual max_pixel_value within the histogram that will not degrade the display to human perceptual visual system. The extended NALU header or SEI message is then inserted into the payload stream in association with the corresponding segment data, and the encoded video data is transmitted (step 604). If the video encoding is incomplete (step 605), another iteration of the process is performed for the pixel data for the next frame, GOP, scene or time interval segment of the video being encoded.
Display adaptation using an extended NAL unit header message based on a perceptually lossless algorithm is exemplified in the above disclosure. Such an algorithm requires the maximum pixel value, set base on perceptual losslessness and specified in the embedded information. However, the principles disclosed are not limited to only such implementation. In another embodiment, any information derived from the video encoder may be embedded as part of the extended NALU header or SEI message to help the display adaptation, such as both minimum and maximum pixel brightness values, or even the histogram distribution.
The present disclosure will make products, such as smartphones and tablets, much more power efficient while reducing the data cost, thus improving the user experience for mobile streaming applications.
While each process flow and/or signal sequence depicted in the figures and described above depicts a sequence of steps and/or signals, either in series or in tandem, unless explicitly stated or otherwise self-evident (e.g., a signal cannot be received before being transmitted) no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions or transmission of signals thereof serially rather than concurrently or in an overlapping manner, or performance the steps or transmission of signals depicted exclusively without the occurrence of intervening or intermediate steps or signals. Moreover, those skilled in the art will recognize that complete processes and signal sequences are not illustrated or described. Instead, for simplicity and clarity, only so much of the respective processes and signal sequences as is unique to the present disclosure or necessary for an understanding of the present disclosure is depicted and described.
Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims.
This application hereby incorporates by reference U.S. Provisional Patent Application No. 61/636,549, filed Apr. 20, 2012, entitled “DISPLAY POWER REDUCTION USING EXTEND NAL UNIT HEADER INFORMATION,” U.S. Provisional Patent Application No. 61/636,543, filed Apr. 20, 2012, entitled “DISPLAY POWER REDUCTION USING EXTEND SEI INFORMATION,” AND U.S. Provisional Patent Application No. 61/636,561, filed Apr. 20, 2012, entitled “PERCEPTUAL LOSSLESS DISPLAY POWER REDUCTION.”
| Number | Date | Country | |
|---|---|---|---|
| 61636549 | Apr 2012 | US | |
| 61636543 | Apr 2012 | US | |
| 61636561 | Apr 2012 | US |
