Cameras capture a scene in the real-world as a static image or video by exposing various capture mechanisms to light. An analog camera captures the scene using analog means, such as a filmstrip, and a digital camera captures the scene using sensors that translate the scene into a digital representation. To facilitate how well the capture mechanisms replicate the scene, a camera regulates how and when to expose the capture mechanisms to light, also referred to as exposure metering. An automatic exposure control (AEC) mechanism automates adjustments to exposure metering that, in turn, affects how well the static image recreates the scene. However, these automated adjustments are using fixed exposure metering which can lead to faulty exposure settings since the fixed exposure metering does not always properly account for the luminance of a scene in its entirety. Thus, it is desirable to have a way to dynamically determine how to adjust exposure settings based upon the various objects in and/or luminance of a scene.
While the appended claims set forth the features of the present techniques with particularity, these techniques, together with their objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
Turning to the drawings, wherein like reference numerals refer to like elements, techniques of the present disclosure are illustrated as being implemented in a suitable environment. The following description is based on embodiments of the claims and should not be taken as limiting the claims with regard to alternative embodiments that are not explicitly described herein.
Various embodiments provide dynamic adjustments to exposure metering used in digital image capture based upon a depth map. A computing device includes at least two image sensors that are synchronized to capture an image or frame of a scene at a same time. Some embodiments, prior to creating a digital image capture, generate a depth map based upon a current frame of a scene that is in view of the image sensors. In turn, the computing device generates weighting values based upon the depth map, and calculates a current frame luma based upon these weighting values. The computing device then calculates settings to adjust exposure metering based upon the current frame luma to improve the digital image capture relative to a digital image capture with fixed exposure metering.
Consider now an example environment in which various aspects as described herein can be employed.
Example Environment
Image capture module includes image sensors 106 that work together in concert to generate a digital image. For example, image sensors 106 include two image sensors that capture respective images. In some cases, each respective image sensor is designed to capture different information, such as color image information, clear or shading image information, raw image data, and so forth. In other cases, the image sensors each capture a respective image in a same format and/or with the same information, but from a different perspective. Images can be stored in various color spaces and representations, such as Red-Green-Blue (RGB), standard Red-Green-Blue (sRGB), Luminance-Blue-Luminance-Red-Luminance (YUV), a color-opponent space with Lightness and color-opponent dimensions (CIE L*a*b), and so forth. These images can also be stored or expressed in any suitable format, such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), bitmap (BMP), Portable Network Graphics (PNG), High-Dynamic-Range Imaging (HDRI), and so forth. These sensors can have various resolutions and be of any suitable types as further described herein. Here, the term “image sensor” generally represents a sensor that is used to capture a corresponding image, and can be a single sensor, or multiple smaller sensors that work together to generate a single image.
Image capture module 104 also includes depth map generator module 108, weighting table generator module 110, and exposure metering control module 112. Among other things, depth map generator module 108 represents functionality that generates relational information about objects in a scene, such as a depth map. To do so, some embodiments of depth map generator module 108 use image information about a scene captured or currently in view of image sensors 106 to generate the relational information about various objects and/locations positioned in scene 114. For example, depth map generator module 108 can use digital image 116 and digital image 118 to generate a depth map. However, other types of information can be used to generate a depth map, such as frame information that generates statistics about a scene in view of the image sensors.
Weighting table generator module 110 represents functionality that dynamically generates a weighting table associated with calculating a frame luma that can be used to adjust exposure settings used by image capture module 104 (e.g., exposure time settings, luminance gain settings). Among other things, a frame luma represents a metric that indicates the luminance or brightness of a scene as seen by image sensors. Weighting table generator module 110 can generate a single weighting table that is used to adjust exposure settings for image capture module 104 as a whole, or generate multiple weighting tables, where each respective weighting table corresponds with a respective image sensor. In some cases, the dynamic generation of a weighting table is based on relational information and/or a depth map generated by depth map generator module 108. A weighting table can be any suitable size, include any suitable number of grid elements, and/or include any suitable weighting values, examples of which are provided herein.
Exposure metering control module 112 represents functionality that automatically adjusts exposure settings used by image sensors 106 and/or image capture module 104, such as exposure time settings and/or luminance gain settings. In some embodiments, exposure metering control module 112 uses the weighting table generated by weighting table generator module 110 to calculate a current frame luma for a current frame or scene in view of image sensors 106. Calculating the current frame luma can include applying weighting information in the weighting table to statistical information generated by image sensors 106. Upon calculating a current frame luma, exposure metering control module 112 adjusts the exposure settings for one or all of image sensors 106.
Environment 100 includes a scene 114 that generally represents any suitable viewpoint or object that an image capture module can visually capture. In this example, each respective image sensor of image sensors 106 captures a respective image that relates to scene 114 such that a first image sensor captures digital image 116 and a second image sensor captures digital image 118. In this example, digital image 116 and digital image 118 are illustrated as capturing different types of information about scene 114, but in alternate embodiments, digital image 116 and digital image 118 capture the same information. When considered together, these dual images can be considered a frame or a captured image of scene 114.
Having described an example operating environment in which various embodiments can be utilized, consider now a discussion of digital image captures in accordance with one or more embodiments.
Digital Image Captures
Image capture mechanisms preserve an image based upon their exposure to light. An analog camera exposes a filmstrip as a way to detect or capture light. Light alters the filmstrip and, in turn, the image can be recovered by chemically processing the filmstrip. In an analog image capture, the filmstrip stores a continuous representation of the light (and corresponding scene). Digital image sensors, too, are exposed to light to capture information. However, instead of an analog representation, digital image sensors generate and store discrete representations of the image.
Consider
The size of a discrete component within a digital image, as well as the number of discrete components, affects a corresponding resolution of the image. For example, image 304 is illustrated as having 9×8=72 discrete components. However, relative to the analog capture represented in image 302, it can be seen that there are inaccuracies in the digital image capture. Given the size of each discrete component, and the uniform nature across the whole of the discrete component, the resultant image lacks details that can be found either in image 302 or original scene 114. By increasing the number of components and reducing the size of the components, the resultant digital image can more accurately capture details and add resolution to the image to more closely resemble the analog version and/or the original captured image. A pixel refers to a singular discrete component of a digital image capture that is the smallest addressable element in the image. Thus, each pixel has a corresponding address and value.
A singular image sensor, such as one of the image sensors in image sensors 106 of
Luma channel information refers to brightness or intensity related to a captured image. For instance, an image or pixel with little to no brightness (e.g., dark or black) would have generally 0% luminance or brightness, while an image or pixel that is has a large amount of light has generally 100% luminance (e.g., bright or white). Depending upon the brightness or intensity, luma information over an image can vary over a range of 0%-100%. Instead of tracking color information, luma information captures light intensity, and can alternately be considered in some manner as a greyscale representation of an image. When a camera captures an image, the intensity or brightness captured in an image can sometimes be controlled through AEC.
These examples are for discussion purposes, and are not intended to be limiting. Further, it is to be appreciated that the technical aspects of digital image capture have been simplified, and is not intended to describe all aspects of digital image capture, Bayer grids, other color formats or filters, and so forth. Having described various principles associated with digital image capture, now consider a discussion of fixed automatic exposure control.
Fixed Automatic Exposure Control
AEC automatically configures exposure metering in an image sensor to achieve a desired brightness or luma value for a particular capture. For example, in scenes that have low lighting, the AEC may adjust exposure time settings and/or luminance gain settings for various sensors to be more sensitive to, or emphasize, the brightness and/or luma, while in scenes that have bright lighting, the AEC may adjust the exposure settings to be less sensitive to, or deemphasize, the brightness. These adjustments help to achieve proper exposure for the scene when the camera captures it, so that the resultant static image more accurately represents the scene than with no adjustments. In some cases, a computing device and/or camera can use a preview or viewfinder function to analyze a scene in order to determine how to configure the AEC. For instance, a preview function can scan a current scene within view of the image sensors (alternately referred to as a frame), and identify current illumination levels (e.g., a high illumination level having bright lighting, a low illumination level having low lighting). These illumination levels of the current scene can then be used determine how to adjust the sensors.
Exposure metering algorithms used by a camera calculate a current frame luma of the current scene or frame based upon statistics generated by the sensors, such as Bayer grid statistical information for a grid of pixels as further described herein. After calculating the current frame luma, the camera compares it to a predefined luma target and determines the difference between the current frame luma and the pre-defined luma target. This difference is the used to obtain exposure metering adjustment information (e.g., exposure time and/or luminance gain for the image sensors) by consulting a predefined sensor-specific exposure table. In turn, the camera then reconfigures the sensor hardware based upon the exposure time and gains to achieve a desired sensitivity to brightness by the sensors. When needed, the camera repeats this process until the current luma frame reaches the luma target, or is within an acceptable tolerance. Thus, AEC helps meter the exposure of light to a capture mechanism based on a current frame luma.
Table 502 illustrates an average-weighted frame. In this example, each element of the grid has a same weight of 1.0. When using the average-weighted exposure metering, the AEC algorithm equally weights the luma statistics generated by the image sensors for a current frame to calculate the current frame luma. Conversely, table 504 illustrates center-weighted exposure metering, where the AEC algorithm applies different weightings to the statistics generated by the image sensors for the current frame. When using center-weighted exposure metering, the center cluster of elements has a higher weighting (illustrated here as 2.40) relative to the outer perimeter of elements (illustrated here as 0.533). Thus, the AEC algorithm gives a center portion of an image more priority or weight in determining an exposure setting for a subsequent image capture. In other words, center-weighted exposure metering adjusts the camera's exposure settings to obtain proper exposure for the center of a scene in the subsequent image capture at the trade-off of degrading the exposure for the perimeter of the scene. While the resultant current frame luma results considers the perimeter elements of the scene, the corresponding statistics of the perimeter elements contribute less to the overall determination of the current frame luma, and thus the exposure metering applied by AEC. Now consider table 506, in which the perimeter elements and/or the statics generated by the corresponding sensors for a current frame have no weighting at all. As can be seen, the spot-weighted exposure metering of table 506 applies a weighting of 0 to the perimeter elements, and a weighting of 16 to the center elements, thus disregarding all statistics generated about the perimeter of the scene.
While table 502, table 504, and table 506 offer options to adjusting a camera's exposure metering, there are drawbacks. To begin, after product launch of a camera, these tables are fixed. For example, the values of the weighting given to the elements, as well as the designation of the areas for different weightings are predefined at product launch and stored on the camera. With respect to spot-weighted exposure metering, a camera can sometimes move the spot to a designated Region-of-Interest (ROI) based upon either a user specifying the ROI or an auto-focus feature identifying an ROI. However, even in the case of spot-weighted exposure metering in which the ROI moves, the ROI region remains fixed in size and shape (e.g., a fixed rectangle). In the real-world, not all objects follow the sizing or the shape of a pre-defined ROI region. These drawbacks associated with fixed AEC can lead to improper exposure metering for an image, thus negatively impacting the quality of a resultant image capture.
Having described various principles associated with fixed AEC, now consider a discussion of dynamic AEC adjustments in accordance with one or more embodiments.
Dynamic Automatic Exposure Control Based on Depth Map
Various embodiments provide dynamic adjustments to exposure metering of an image capture module and/or image capture device based upon a depth map. Image capture devices that include synchronized image sensors can capture multiple images of a scene and/or frame information at a same time. By having synchronized images and/or frame information, various embodiments generate depth and/or distance information for different objects within the scene that can then be used to dynamically modify exposure metering by modifying various exposure settings of the image sensors as further described herein.
A depth map provides distance information for different objects or location within a scene. In some cases, this distance information represents the distance of an object from a particular viewpoint or frame of reference. However, this distance information can additionally include, or be used to extract, relational information between the different objects. To further illustrate, consider
To illustrate, consider a point “X” within a scene, where “X” is at an arbitrary distance from a camera (or other type of computing device) that includes two image sensors (also known as a dual camera). In capturing this scene (and “X”), the camera generates a first image and a second image using a first image sensor and a second image sensor, respectively. Since there are two images sensors, and they do not share a same physical space, the first image and the second image have differing views of the scene, where each viewpoint of the view derives itself from the positioning of the respective image sensor. In light of this, the first location of “X” in the first image differs from the second location of “X” in the second image. Generally, a depth of a point or object in a scene is inversely proportional to the difference in distance of the image points and their respective camera centers. In applying this to the dual image capture that includes “X”, the depth of “X” can be determined using knowledge of the first location of “X” relative to the center of the first image sensor, and the second location of “X” relative to the center of the second image sensor. In some cases, determining the depth of “X” alternately or additionally uses information pertaining to the relative positioning of the image sensors (e.g., the distance between a first image sensor and a second image sensor).
Here, depth map 604 corresponds to relational positioning and/or depth information of the various objects or locations associated with image 602. In depth map 604, an object or location with a darker hue corresponds to location that is further away from the image sensor than an object or location with a lighter hue. Scale 606 also illustrates this, where the top of the scale has a white hue and gradually transitions to a black hue at the bottom. The variation of shades that transition between the lightest hue on scale 606 to the darkest hue on scale 606 generally indicate the varying distances or depths in which objects reside in a scene. Thus, lighter hues correspond to depth distances closer to the image sensor (e.g., less depth), darker hues correspond to depth distances further from the image sensor (e.g., more depth or distance), and the various hues in between the lightest hue and the darkest hue correspond to depth distances in between the closest and furthest depths, respectively. As can be seen in image 602, sign 608-1 is positioned in the foreground of image 602, sky 610-1 is positioned in the background of image 602, and trees 612-1 are positioned intermediate and in between the foreground and background. In depth map 604 and relative to one another, sign 608-2 has the lightest hue, sky 610-2 has the darkest hue, and trees 612-2 have a hue that is in between the lightest hue and the darkest hue. For illustrative purposes, depth map 604 represents a visual representation of distance and/or depth information stored within the depth map using a greyscale image.
Once the camera generates a depth map, various embodiments dynamically generate weightings used calculate a current frame luma, and subsequently configure exposure metering by making adjustments to various exposure settings of image sensors (e.g., exposure time settings, luminance gain settings). Consider
Using the three levels of categorization, when a depth value is categorized as being foreground, the camera assigns a corresponding element in weighting grid 702 a value of 1.0. Applying this to sign 608-2 of
As further described herein, a grid element sometimes corresponds to multiple pixels and/or a grid of pixels. In turn, each respective pixel can have its own depth value. When a grid element corresponds to multiple pixels, each pixel may generate a respective depth value, thus producing multiple depth values for a respective weighting grid element. In such a case, the multiple depth values and/or multiple weighting values can be interpolated to generate a resultant weighting value. Consider weighting grid element 710 that has an assigned weighting value of 0.9. Assume for this example that the corresponding depth map for weighting grid element 710 has two depth values: a first depth value that is classified as a foreground value (with a weighting of 1.0) and a second depth value that is classified as an intermediate depth value (with a weighting of 0.8). Here, the camera generates the subsequent weighing value of 0.9 by interpolating the two weighting values, and assigns weighting grid element 710 the subsequent (interpolated) value. Thus, a weighting grid can include interpolated weighting values.
While weighting grid 702 represents a weighting grid in which foreground objects are given a higher weighting or priority than background objects, other priority assignments can be used. For example, in some cases, a weighting grid based off a depth map assigns a higher priority to background objects relative to foreground objects, or assign intermediate objects a higher priority than foreground and background objects. In some cases, a camera has default weighting priorities, such as foreground objects having a higher priority than background objects, that a user can subsequently change through a User Interface (UI). Alternately or additionally, a user can define a ROI to have a higher priority, where the user identifies a center position for the ROI, and a depth map is used to dynamically identify a size and shape of the ROI. Accordingly, the camera can assign weighting values based upon default priority information or obtain priority information from the user. Some embodiments base priority information on object size, such as by assigning a higher priority and/or weighting to larger objects and decreasing the weighting as object sizes get smaller, or assigning a higher weighting and/or priority to smaller objects than larger objects. Thus, various embodiments allow for dynamic configuration of priority assignments and/or weighting values.
Consider
Step 802 obtains at least two current frames via at least two image sensors. This can include obtaining statistics from the at least two image sensors, such as Bayer Grid statistical information. In some cases, when an image sensor has multiple smaller sensors, each smaller sensor generates a corresponding statistic. Obtaining the current frames sometimes includes obtaining multiple image captures of a scene, where each respective current frame originates from a respective image sensor.
Responsive to obtaining the current frames, step 804 generates a depth map based on the current frames. Here, the depth map includes relational information about objects, points, or locations within the current frames, such as distance information. For example, to generate the depth map, some embodiments utilize at least two synchronized images of a same scene to discern distance information, as further described herein. However, alternate forms of relational information can be generated as well.
Step 806 generates a weighting table based on the depth map or other relational information. For instance, a camera can have a predetermined number of levels used to assign weightings, such as a predetermined number of distance classifications (e.g., foreground, intermediate, background). Any suitable number of levels can be utilized. In turn, the camera analyzes and categorizes the depth map distances that correspond to respective regions and/or pixels of the current frame into one of the levels, and then assigns the respective grid element of the weighting table a weighting value. A grid element can correspond to a single pixel or multiple pixels. When a grid element corresponds to multiple pixels, some embodiments interpolate the weighting value assigned to the grid element as further described herein. The weighting values that are assigned to the respective levels can be determined using default priority information, or user-defined priority information (e.g., ROI information, object information, foreground or background selection, etc.).
Step 808 calculates a current frame luma based upon the weighting table. For instance, some embodiments apply the values in the weighting table to respective statistical information generated by image sensors. Responsive to calculating the current frame luma, step 810 adjusts exposure settings associated with the image sensors, such as exposure time settings and/or luminance gain settings.
Step 812 determines if the current frame luma is at a predefined target luma value, or close enough to the target luma value. Some embodiments determine that the current frame luma is close enough to the predefined target luma if it falls within an acceptable tolerance to the predefine target luma. In some cases, a camera or computing device determines the difference between the current frame luma and the predefined luma target as a way to identify how to reconfigure the sensor hardware. Reconfiguring the sensor hardware can include modifying and/or reconfiguring exposure time and luminance gains to achieve a desired sensitivity to brightness based upon this difference. If the current frame luma fails to be at or close enough to the predefined target luma, the method proceeds to step 802 and repeats steps 802-812 in order to readjust the exposure settings to, in turn, generate a current frame luma that is within the predefined tolerance. If the current frame luma is considered to be at, or close enough to, the predefined target luma, the method proceeds to step 814.
Step 814 generates an image capture using the adjusted exposure settings that impact the camera's exposure metering (e.g., exposure time settings, luminance gain settings). This can occur automatically after the final adjustments to the exposure metering occurs, or can occur after receiving user input to generate the image capture.
While the method described in
Having considered a discussion of a dynamic exposure metering based upon a depth map, consider now a discussion of empirical data generated in accordance with various embodiments described herein.
Empirical Test Data
Fixed weighting tables used in exposure metering oftentimes fails to adjust the exposure metering based upon real-world objects and lighting in a particular scene or frame. In turn, this can produce image captures with poor quality with partial regions of the image being underexposed or overexposed. Conversely, exposure metering based upon a depth map accounts for the sizes, shapes, and locations (e.g., depth) of the various object. In turn, depth map-based exposure metering improves the image quality of an image capture relative to those generated with fixed weighting exposure metering. To test the difference in image quality between fixed weighting AEC and depth map-based AEC, a camera captured a same scene using different exposure metering algorithms: first using fixed weighting exposure metering, then using dynamic exposure metering based upon a depth map.
Now consider image 906 that displays an image capture generated with dynamic weighting generation based upon depth map as further described herein. Similar to that of image 902, outdoor window 904 is positioned in the background, causing the same backlight condition. However, since the exposure metering is weighted and based upon objects and/or distances included in the scene, flower vase 908 is no longer obscured. Instead, the settings applied for the exposure metering provides enough exposure to the corresponding region to make flower vase 908 visible, as well as other details about the room that were previously obscured. In this example, the weighting priority assigns higher priority to foreground objects and results in an image capture with less underexposed regions. Thus, in terms of clearly capturing objects within a scene, image 906 with dynamic weighting generation based upon a depth map provides an improved image over image 902.
Having considered a discussion of an example test case that employed dynamic exposure metering based upon a depth map, consider now a discussion of an example device which can include dynamic exposure metering based upon a depth map in accordance with various embodiments described herein.
Example Device
Electronic device 1000 includes processor system 1002 (e.g., any of application processors, microprocessors, digital-signal processors, controllers, and the like) or a processor and memory system (e.g., implemented in a system-on-chip), which processes computer-executable instructions to control operation of the device. A processing system may be implemented at least partially in hardware, which can include components of an integrated circuit or on-chip system, digital-signal processor, application-specific integrated circuit, field-programmable gate array, a complex programmable logic device, and other implementations in silicon and other hardware. Alternately or in addition, the electronic device can be implemented with any one or combination of software, hardware, firmware, or fixed-logic circuitry that is implemented in connection with processing and control circuits, which are generally identified as processing and control 1004. Although not shown, electronic device 1000 can include a system bus, crossbar, interlink, or data-transfer system that couples the various components within the device. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, data protocol/format converter, a peripheral bus, a universal serial bus, a processor bus, or local bus that utilizes any of a variety of bus architectures.
Electronic device 1000 also includes one or more memory devices 1006 that enable data storage, examples of which include random access memory (RAM), non-volatile memory (e.g., read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and a disk storage device. Memory devices 1006 are implemented at least in part as a physical device that stores information (e.g., digital or analog values) in storage media, which does not include propagating signals or waveforms. The storage media may be implemented as any suitable types of media such as electronic, magnetic, optic, mechanical, quantum, atomic, and so on. Memory devices 1006 provide data storage mechanisms to store the device data 1008 and other types of information or data. In some embodiments, device data 1008 includes digital images. Memory devices 1006 also provide storage for various device applications 1010 that can be maintained as software instructions within memory devices 1006 and executed by processor system 1002.
To facilitate image capture, electronic device 1000 includes image capture module 1012. Here, portions of image capture module 1012 reside on memory devices 1006: depth map generator module 1014, weighting table generator module 1016, and exposure metering control module 1018, while other portions of image capture module 1012 are implement in hardware: image sensors 1020. While illustrated here as residing on memory devices 1006, alternate embodiments implement depth map generator module 1014, weighting table generator module 1016, and/or exposure metering control module 1018 using varying combinations of firmware, software, and/or hardware
Among other things, depth map generator module 1014 generates relational information about objects in a scene, such as a depth map, by using captured digital images or frame information about a scene that is in view of image sensors 1020. Weighting table generator module 1016 dynamically generates a weighting table used to adjust exposure settings, such as exposure time and luminance gain settings associated with image sensors 1020. Exposure metering control module 1018 adjusts exposure metering associated with image sensors 1020 based upon the weighting table generated by weighting table generator module 1016. In some embodiments, adjusting the exposure metering includes adjustments to exposure time and/or luminance gain settings associated with image sensors 1020.
Image sensor(s) 1020 represent functionality that digitally captures scenes. In some embodiments, each image sensor included in electronic device 1000 captures information about a scene that is different from the other image sensors, as further described above. For example, a first image sensor can capture a color image using Bayer techniques, and a second image sensor can capture clear images. The sensors can be individual sensors that generate an image capture, or include multiple smaller sensors that work in concert to generate an image capture.
It is to be appreciated that while electronic device 1000 includes distinct components, this is merely for illustrative purposes, and is not intended to be limiting. In view of the many possible embodiments to which the principles of the present discussion may be applied, it should be recognized that the embodiments described herein with respect to the drawing figures are meant to be illustrative only and should not be taken as limiting the scope of the claims. Therefore, the techniques as described herein contemplate all such embodiments as may come within the scope of the following claims and equivalents thereof.