The present disclosure generally relates to the field of electronics. More particularly, some embodiments of the invention relates to fine grained power gating of camera image processing.
As mobile computing devices become more common place, it is imperative to reduce power consumption in such devices as much as possible while maintaining usability. More particularly, since mobile computing devices generally rely on batteries with a limited life, the amount of power consumed for various operations needs to be closely guarded to increase battery life, as well as addressing thermal limits.
Furthermore, transistors optimized for performance tend to be more leaky. In order to address rapidly increasing market requirements, and accordingly higher performance needs, such transistors become a natural choice for an image signal processor. Hence, it becomes imperative to reduce leakage power through innovations other than process technology to stay competitive.
The detailed description is provided with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of various embodiments. However, some embodiments may be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the particular embodiments.
Some embodiments may address power consumption of an Image Signal Processor (ISP) during still or video capture. In an embodiment, leakage power of an ISP is reduced by partitioning the ISP hardware into separate (e.g., two) power domains that may be independently powered on and off. In one embodiment, one or more frame or ring buffers may be used to process data in a burst to reduce power leakage by an image processing portion of an ISP during image capture and buffering operations. Moreover, the techniques discussed herein may be applied to any type of a ISP device, including for example mobile devices (such as a mobile phone, a laptop computer, a personal digital assistant (PDA), an ultra-portable personal computer, tablet, etc.) or non-mobile computing devices (such as a desktop computer, a server, etc.).
Furthermore, wireless or wired communication channels may be utilized for transfer of data between various components of an ISP system. The wireless communication capability may be provided by any available wireless connection, e.g., using a Wireless Wide Area Network (WWAN) such as 3rd Generation (3G) WWAN (e.g., in accordance with International Telecommunication Union (ITU) family of standards under the IMT-2000), Worldwide Inter-operability for Microwave Access (“WiMAX, e.g., in accordance with Institute of Electrical and Electronics Engineers (IEEE) 802.16, revisions 2004, 2005, et seq.), Bluetooth® (e.g., in accordance with s IEEE Standard 802.15.1, 2007), Radio Frequency (RF), WiFi (e.g., in accordance with IEEE 802.11a, 802.11b, or 802.11g), etc. Also, the wired communication capability may be provided by any available wired connection, e.g., a shared or private bus (such as a Universal Serial Bus (USB)), one or more (unidirectional or bidirectional) point-to-point or parallel links, etc.
As illustrated in
To improve battery life performance for mobile devices (such as Smartphones or tablet SoC (System on Chip), into which these ISPs are integrated), there needs to be very efficient power management. Power management may be done to manage active power, but power management may also be applied to lower the leakage power as it is becoming increasingly significant due to constantly increasing size of ISPs with each generation. To this end, some embodiments address leakage power and include novel techniques to reduce the leakage power consumption.
Generally, camera image sensors do not include frame buffers, and without being able to buffer the pixels after exposure, sensors send the pixel data out as and when the pixels are exposed. Furthermore, the excitation from the light sources is continuous and the integration happens over the entire frame time. As a consequence, these sensors transmit pixels over the entire frame period. This arrival rate at the ISPs has an impact on leakage power if the processing capacity exceeds the input pixel rate. For instance, if the ISP is designed to process a full frame of M megapixels, then for a decimated frame, say, by a factor of two in each X, Y, the ISP is required only 25% of the time. But in practice, since the pixel arrival rate is spread throughout the frame, the ISP has no opportunity to be powered off as shown in
In an embodiment, a similar optimization is also applicable during video mode. More particularly, most ISPs are designed to process multi-megapixel full image frames during still capture. These ISPs will require a fraction of the time to process a video frame, even for 1080p video as the frame size is only 2 Mega pixels. During video processing, with conventional implementation, ISPs remain ON burning leakage power, e.g., for more than 70% of the time, in excess of the duration ISPs are actually required for processing the video frame.
In one embodiment, two separate power domains (also referred to herein as partitions) may be used for an ISP, one (power domain A) for the ISP receiver 104 circuitry and the other (power domain B) for the ISP processor 106 circuitry. Each domain may be supplied by separate power rails, isolation circuits and/or clock signals (e.g., clk1 and clk2) to allow for power gating of each domain independently. In one embodiment, clocks clk1 and clk2 may be derived from a same clock signal (e.g., by using a clock divider). As discussed herein, power gating generally refers to controlling the level of operations in a given domain (e.g., by controlling the frequency of a clock supplied to the domain). In an embodiment, to power off a domain, its respective clock may be turned off. Alternatively, the clock may be slowed down instead of totally shutting it off (e.g., to reduce the overhead/delay associated with bringing the shut off circuits online and operational or to otherwise reduce power consumption). The ISP receiver 104 area in comparison to the ISP area may be insignificant in an embodiment. So, when the pixels are being acquired from the sensor 102 into the frame buffer 107 in memory 108, only the ISP receiver 104 circuitry is ON. In other words, only the ISP receiver 104 circuitry needs to be ON for the entire frame time. The processor portion of the ISP 106 on the other hand remains OFF when the pixels are being acquired and is turned ON for the duration when it has to process the already stored frame.
Accordingly, in one embodiment, the ISP may include two partitions. The ISP processor partition may be put into a low power consumption state during a first time period when the ISP receiver partition is receiving (and storing the sensor data in the memory). In various embodiments, the low power consumption state may include a partial power consumption state or an complete power off state. During a second time period (e.g., after the captured image sensor data is stored in memory), the ISP processor partition may process the stored image data. Also, the ISP receive partition may receive and store additional image sensor data in the memory during a third time period that overlaps with or follows the second time period. Further, since ISP area may be a significant contributor to the leakage power, power leakage is efficiently managed by the aforementioned technique. For instance during viewfinder, the ISP leakage may be reduced close to 90% and for video leakage may be lowered by close to 70%. In some embodiments, a finer power domain partitioning of ISP processors may be provided to further reduce power consumption, e.g., power gating of even the ISP processor portion components. By having multiple ISP processor power domains, there is further opportunity for reducing power leakage when the ISP is under-utilized.
As the implementations of camera systems vary, some embodiments of the invention further address two alternate implementations. In the first alternate embodiment, the sensor 102 sends the full frame even during viewfinder of video. This is done as sometimes the sensor binning or scaling support is inadequate or of low quality. In this alternate embodiment, the ISP Rx 104 includes a scaler engine/block/logic. This scaler logic may downscale the incoming full frame to the target frame resolution, be it viewfinder frame size or video frame size. By doing so, only the required data is written into the frame buffer 107 in memory 108, e.g., to reduce storage requirements, memory/interconnect bandwidth usage, power consumption of memory/interconnect, etc. The next embodiment might choose to not only acquire the full frame from the sensor 102 but also to store the full RAW Bayer frame in frame buffer 107. Generally, Bayer format refers to a color space associated with arrangement of an array of color filters of Red, Green, and Blue (RGB) on a grid of photo sensors used in some digital image sensors. In some embodiments, the ISP 106 may convert the image sensor data from an RGB color space to a YUV (Luminance-Bandwidth-Chrominance) color space, e.g., prior to storing the data in the memory 108 or otherwise for chroma/luma modifications/corrections. For such embodiments, the scaler logic may be present in the ISP Rx 104 partition (power domain A) such that when the frame is being processed it may first be downscaled and processed according to the requirement of the target resolution before storing the processed data in the memory 108.
As shown in
Delaying the processing of frames results in a lag from the time the user captures an image/video to the time the user actually sees the captured image/video on a display. This delay may be acceptable up to two to three frames in typical conditions to maintain a good user experience.
The ISP architecture and techniques described above may be employed in various types of computer systems (such as the systems discussed with reference to
Furthermore, the operations discussed with reference to
A chipset 506 may also communicate with the interconnection network 504. The chipset 506 may include a graphics and memory control hub (GMCH) 508. The GMCH 508 may include a memory controller 510 that communicates with a memory 512. The memory 512 may store data, including sequences of instructions, that may be executed by the CPU 502, or any other device included in the computing system 500. In one embodiment of the invention, the memory 512 may include one or more volatile storage (or memory) devices such as random access memory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), or other types of storage devices. Nonvolatile memory may also be utilized such as a hard disk. Additional devices may communicate via the interconnection network 504, such as multiple CPUs and/or multiple system memories.
The GMCH 508 may also include a graphics interface 514 that communicates with a display device 516. In one embodiment of the invention, the graphics interface 514 may communicate with the display device 516 via an accelerated graphics port (AGP) or PCIe. In an embodiment of the invention, the display 516 (such as a flat panel display) may communicate with the graphics interface 514 through, for example, a signal converter that translates a digital representation of an image stored in a storage device such as video memory or system memory into display signals that are interpreted and displayed by the display 516. The display signals produced by the display device may pass through various control devices before being interpreted by and subsequently displayed on the display 516.
A hub interface 518 may allow the GMCH 508 and an input/output control hub (ICH) 520 to communicate. The ICH 520 may provide an interface to I/O device(s) that communicate with the computing system 500. The ICH 520 may communicate with a bus 522 through a peripheral bridge (or controller) 524, such as a peripheral component interconnect (PCI) bridge, a universal serial bus (USB) controller, or other types of peripheral bridges or controllers. The bridge 524 may provide a data path between the CPU 502 and peripheral devices. Other types of topologies may be utilized. Also, multiple buses may communicate with the ICH 520, e.g., through multiple bridges or controllers. Moreover, other peripherals in communication with the ICH 520 may include, in various embodiments of the invention, integrated drive electronics (IDE) or small computer system interface (SCSI) hard drive(s), USB port(s), a keyboard, a mouse, parallel port(s), serial port(s), floppy disk drive(s), digital output support (e.g., Digital Video Interface (DVI)), High Definition Multimedia Interface (HDMI), or other devices.
The bus 522 may communicate with an audio device 526, one or more disk drive(s) 528, and a network interface device 530 (which is in communication with the computer network 503). Other devices may communicate via the bus 522. Also, various components (such as the network adapter 530) may be coupled to the GMCH 508 in some embodiments of the invention. In addition, the processor 502 and the GMCH 508 may be combined to form a single chip. In an embodiment, the memory controller 510 may be provided in one or more of the CPUs 502. Further, in an embodiment, GMCH 508 and ICH 520 may be combined into a Peripheral Control Hub (PCH).
Furthermore, the computing system 500 may include volatile and/or nonvolatile memory (or storage). For example, nonvolatile memory may include one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), a disk drive (e.g., 528), a floppy disk, a compact disk ROM (CD-ROM), a digital versatile disk (DVD), flash memory, a magneto-optical disk, or other types of nonvolatile machine-readable media that are capable of storing electronic data (e.g., including instructions).
Furthermore, the operations discussed with reference to
As illustrated in
In an embodiment, the processors 602 and 604 may be one of the processors 502 discussed with reference to
At least one embodiment of the invention may be provided within the processors 602 and 604. Other embodiments of the invention, however, may exist in other circuits, logic units, or devices within the system 600 of
The chipset 620 may communicate with a bus 640 using a PtP interface circuit 641. The bus 640 may communicate with one or more devices, such as a bus bridge 642 and/or I/O devices 643. Via a bus 644, the bus bridge 642 may communicate with other devices such as a keyboard/mouse 645, communication devices 646 (such as modems, network interface devices, or other communication devices that may communicate with the computer network 503), audio I/O device 647, and/or a data storage device 648. The data storage device 648 may store code 649 that may be executed by the processors 602 and/or 604.
In various embodiments of the invention, the operations discussed herein, e.g., with reference to
Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least an implementation. The appearances of the phrase “in one embodiment” in various places in the specification may or may not be all referring to the same embodiment.
Also, in the description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. In some embodiments of the invention, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements may not be in direct contact with each other, but may still cooperate or interact with each other.
Thus, although embodiments of the invention have been described in language specific to structural features and/or methodological acts, it is to be understood that claimed subject matter may not be limited to the specific features or acts described. Rather, the specific features and acts are disclosed as sample forms of implementing the claimed subject matter.