Patent Application
20080239119
The disclosed embodiments relate generally to the field of image capturing system and technique.
Conventional camera technology rely on photo-capacitors that develop charge in response to receiving light. Such photo-capacitors are typically tied or used in conjunction with color filters that make the individual element sensitive to light from a particular range in the spectrum. Each photo-capacitor develops a voltage from exposure to light, and the voltage is a function of the intensity of light captured by that photo-capacitor. Often, voltage from individual photo-capacitors is used to interpret a value for that photo-capacitor. This value reflects the intensity of light captured by the corresponding photo-capacitor. Typically, the voltage is converted into a digital value by an analog-digital converter.
By using voltage values from the photo-capacitors, conventional techniques have some limitation in the range of light intensity that can be captured on one image. When exposed to light, photo-capacitors develop charge at a rate that is dependent on the intensity of the light being received. For this reason, photo-capacitors that capture light from a bright object develop charge more quickly than those photo-capacitors that capture light from a dark object. When a bright object is present in a picture that has otherwise normal or less bright surrounding objects, conventional approaches provide that the camera must either limit the exposure of light to capture the bright object or overexpose the bright object to capture the surrounding objects. Often, the camera captures the bright object and the remainder of the picture is dark.
Additionally, cameras typically require use of analog-digital converters in order to convert voltage values from photo-capacitors. Such converters are susceptible to noise, and add intricacies and cost to the device.
Embodiments described herein provide an image capturing system, device and technique which use time-based information about the charge state of individual pixels of a pixel grid on which light is cast, in order to create a digital picture. In particular, embodiments described herein define the charge state of individual pixels based on a charge response to light exposure, when, for example, a camera shutter is opened. When an image is to be captured and light exposure is initiated (e.g. when the shutter is opened), the individual pixels that comprise the grid on which light is cast are repeatedly sampled to determine the instance, during the duration of light exposure (e.g. while the shutter is opened), when the state of each pixel changes. This state information about each pixel, including the instance when each pixel changed states, may then be compressed or otherwise correlated into pixel values from which an image can be created.
According to an embodiment, the charge state may be binary in nature, in that the state is one of the pixel providing or not providing a charge response that exceeds a designated threshold.
As embodiments described herein collect state information about individual pixels, one or more embodiments eliminate a need for an analog-to-digital converter. Rather, state information may be collected and used in digital form.
Moreover, one or more embodiments provide that the use of time-based state information enables an image capturing system to collect sufficient light from all objects in a scene so that the image displays all objects with brightness that is accurate. In particular, one or more embodiments enable a pixel grid to capture and reflect the brightness level from all objects in a scene, regardless of whether there is an object or portion of the scene that is significantly brighter than another portion. In this way, light reflected from each object in a scene may be captured without influence or affect of the brightness of other objects in the scene.
According to an embodiment, an image-capturing device includes a lens, a grid of light-sensitive elements, a sampling mechanism, and an image generation module. The lens is configured to be triggerable to open for a given duration. The grid is positioned so that when the lens is open, light is cast onto the grid. Each element of the grid is configured to provide a charge response to light. The sampling mechanism is coupled to the grid to repeatedly sample the grid during the duration by detecting whether the charge response of individual elements that comprise the grid exceeds a designated threshold level. The sampling mechanism records information that is indicative of a time in the duration in which the charge response of individual elements exceeds the designated threshold level. The image creation module generates an image corresponding to the scene using the information recorded by the sampling mechanism.
As used herein, the term “pixel” is intended to mean a light-sensitive element. A pixel may include or be in the form of a photo-capacitor.
Embodiments described herein may be implemented or used on any device equipped to capture images, including digital cameras and multi-functional devices, such as cellular phones and messaging devices that are equipped with cameras.
One or more embodiments described herein provide that methods, techniques and actions performed by a computing device are performed programmatically, or as a computer-implemented method. Programmatically means through the use of code, or computer-executable instructions. A programmatically performed step may or may not be automatic.
In an embodiment, the light-sensitive elements 132 of the grid 130 correspond to photo-capacitors. The elements 132 may be equipped with or coupled to filters to make them sensitive to color (red, green and blue). The elements 132 may sometimes be referred to as pixels.
During the exposure period, grid interface 140 interfaces with the grid 130 to determine time-based information about the charge response of individual elements 132 that comprise the grid. According to an embodiment, the grid interface 140 interfaces with individual elements 132 of the grid to determine information about when individual elements provide a charge response that meets or exceeds a designated threshold level. The charge response may correspond to an increase or decrease of charge stored or held by that element, resulting primarily from the individual elements 132 being exposed to light.
In an embodiment, the grid interface 140 includes hardware that scans the grid 130 repeatedly during the exposure duration. In one embodiment, the grid interface 140 includes a mechanism to compare a charge value of each element with a designated threshold value. With each scan of the grid 130, the grid interface 140 compares the determined charge value at the instance of the scan with the designated threshold value. The grid interface 140 can mark the instance when the element 132 is first scanned and determined to have a charge response that meets or exceeds the designated threshold level. The grid interface 140 may determine or return time-based, state information (“time-based information 135”) that reflects one or more of the following: (i) at each given instance when the shutter is opened and the grid is scanned, whether individual elements 132 that comprise the grid 130 had a charge response to the light exposure that met or exceeded a designated threshold level, or (ii) the particular instance that determined when each individual element 132 had a charge response to the light that met or exceeded the designated threshold level.
The time-based information 135 may be stored (temporarily or otherwise) with memory resources 150 and used by processing resources 160 to create a digital image 170. The memory resources 150 may correspond to buffers or other mediums in which information from the grid interface 140 is temporarily stored to permit the processing resources 160 to create the image.
The processing resource 160 may perform any one of many algorithms to convert the time-based information 135 into the digital picture 170. In one embodiment, for example, the processing resource 160 may correlate an instance when each element 132 of the grid 130 had a charge response to a color and/or luminescent value for that element. For example, under one implementation, the algorithms performed by the processing resource 160 include determining a time t that marks when each element 132 changed state during the time of exposure. The processing resource 160 then correlates the time t with a value for that element. As individual elements 132 are equipped to absorb light from a designated spectrum (of red, green or blue), the value of each element 132 may reflect color and intensity or illuminosity. In another embodiment (such as shown by
Initially, in a step 210, a state is defined for individual elements 132 (
Event 202 marks the instance of time T=0, when light is cast onto the grid 130 from a scene of which the image is to be captured. This may correspond to a shutter of a camera device being opened. Subsequently, step 220 provides that the state of individual elements 132 of the grid 130 is checked repeatedly. In an embodiment such as described with
Step 220 may be performed until event 204, when the exposure to light is over, as marked by T=F (i.e. shutter closed). During the period of exposure, step 220 may be performed at each instance (i), and at a frequency of n. As described elsewhere in this application, the frequency in which step 220 is performed for each element in the duration between events 202, 204 may range in order of 102 to 108. (although greater magnitudes are also possible).
In one embodiment, step 220 provides that the state of all elements 132 in the grid 130 is checked at each instance. In another embodiment, the state of some elements 132 in the grid is checked. For example, only those elements that have yet to change state may be checked. As part of step 220, information that is indicative of when each element 132 in the grid 130 changes state is recorded. This may correspond to recording the state of all elements in the grid at each instance (i), or recording for each element the value of i just before and/or after the state of that element changed.
Among other benefits, the conversion of the time-based information 135 to the digital picture 170 may be created in this way without the need for performing analog-to-digital conversions. Moreover, using time-based information 135, the image 170 may maintain and display accurate and fully lighted details of different objects that in actuality have significant disparity in brightness levels. Under conventional image capturing techniques, such images often result in pictures with bright objects making other objects in the image much darker than the objects actually were. This problem is avoided by an embodiment such as described with
One or more embodiments include a sampling process 310 and an image creation process 320. Under one or more embodiments, each process may be performed by a corresponding module. The sampling process 310 samples elements 132 of the grid 130 repeatedly in a duration marked by the event 332 of the shutter opening and the event 334 of the shutter closing. The sampling process 310 may perform the step of repeatedly checking the state of individual elements (as described by step 320 of
According to one embodiment, the output of the sampling process 310 is time-based state information 335. This information 335 records, at numerous instances between the open and close shutter events 332, 334, whether individual elements 132 have provided a charge response that meets or exceeds the designated threshold level. In one embodiment, the state of each element is recorded at each instance when the grid is sampled, regardless of whether or not that element has provided the requisite charge response. In another embodiment, the information 335 indicates the instance between the shutter events 332, 334 when individual elements are detected to provided the requisite charge response (i.e. the instance when the element 132 changed state).
The sampling process 310 passes the time-based state information 335 to the image creation process 320. In an embodiment such as shown, the image creation process 320 is performed after the event 334 of the shutter closing. At this instance, all elements 132 may be assumed to have provided the requisite charge response.
The image creation process 320 may follow and/or be performed concurrently with the sampling process 310. The image creation process 320 may generate a picture using the time-based information 335. The creation of the picture is signified by the event 336. In one embodiment, the image creation process 320 is aware of the range of color values and/or the amount of illuminance (which may be quantified by units of LUX or lumens/square meter) that can be recorded from each element 132 of the grid 130, and correlates the time-based information 335 of each element to a color and/or illuminance value for that element 132. For example, bright objects may trigger a relatively quick charge response from some elements 132 of the grid 130 that record reflected light from that object. In contrast, elements 132 that record light from darker objects in the same scene may require a relatively longer time to trigger the charge response. The time-based information 335 enables the image creation process 320 to determine the amount of light each element records from the scene, independent of the illuminance levels experienced by elements that record light reflected from bright or very bright objects.
In one embodiment, the sampling process 320 is configured to sample the grid 130 at a frequency that is sufficient to detect an instance before and after when a cluster of elements 132 provide the charge response that meets the designated threshold level when exposed to light that has an illuminance value in excess of some maximum value. Depending on the frequency of sampling, the maximum value can range, for example, from 15,000 LUX to 20,000 LUX. At the same time, other elements 132 may provide the charge response that meets the designated threshold level when exposed to light that has an illuminance value that is less than some minimum value. Depending on the length of the shutter speed, the minimum value may be in the range of 5-1000 LUX. With adequate sampling speed and sufficiently long shutter time, it is possible to record and adequately reflect light from two objects in a scene that have extreme disparity in illuminance. For example, a picture of a scene may contain both the sun and a person standing in shade. Under conventional approaches, a camera may generate the image to reflect the brightness from the sun, but the camera would also have to quicken the shutter speed to prevent the brightness of the sun from being whitewashed in the image. In this conventional approach, not enough light would be recorded from the darker objects to fully reflect the illuminance levels of those objects. The result is a picture that has the sun brightly shown, and the rest of the picture dark. In contrast, with the use of time-based information, the image creation process 320 can create an image that contains objects with disparate illuminance levels. This is because the elements 132 are being scanned to determine when they provide the requisite charge level. This allows bright objects and darker ones to be captured in an image together, where the brightness of each object in the image reflects the illuminance value of the corresponding image in the scene, and more specifically, the brightness of a darker object in the image is not limited by the presence of the bright object in the scene.
According to an embodiment, the device 400 includes a shutter mechanism 405, a lens 410, a grid 420 comprising pixels 422, a sampling mechanism 430, and an image creation module 440. In operation, shutter mechanism 405 may expose the lens 410 to light 403 reflected from a scene 402. For a duration T (corresponding to when the shutter 405 is opened), this light may be captured on the grid 420. The pixels 422 may correspond to photo-capacitors that develop charge when exposed to light. Each pixel 422 may also include a color filter so that the pixel is predominantly sensitive to light from a specific range.
According to an embodiment, the sampling mechanism 430 repeatedly samples the grid 420 at individual instances t, where n*t=N, and n is the sample rate. With each instance of sampling the grid 420, the sampling mechanism 430 receives pixel state information 435, represented by psi(t). The pixel state information 435 includes information that is specific to individual pixels 422. As described with an embodiment of
Time-based pixel information 445 is provided by the sampling mechanism 430 to the image creation module 440. The time-based pixel information 445 may correlate or be based on the pixel state information 435. For example, the pixel state information 435 may be filtered or processed to reflect timing of when individual pixels 422 changed state (as to charge response) when the shutter was open. The image creation module may correlate a pixel value 448 to each pixel based on the time-based information 445 for that pixel. The pixel value 448 may reflect a brightness or illuminance, as well as color (depending on the filter associated or included with the particular pixel). The result is the creation of the digital image 450.
The element 532 is provided a switch 510 that extends a reference voltage (Vref) to the element 532. The switch 510 may remain closed until the shutter is opened, in which case the switch 510 becomes open. In an implementation shown, Vref is greater than a designated threshold level. When the shutter and switch 510 are simultaneously opened, the element 532 is exposed to light and develops charge from light exposure. In an embodiment, the element 532 is arranged or configured with other elements to discharge Vref on node 520 when the switch 510 is opened and charge from light is developed. The opening of the switch 510 may coincide with the event of the shuttering opening or other exposure to light. In one embodiment, the element 532 is oriented or configured so that its rate of discharge at node 520 is dependent on its exposure to light. In this way, the element 532 discharges Vref at a rate that is dependent on the intensity of light received by that element. At some point, sufficient discharge of Vref occurs that the voltage at node 520 is less than or equal to a designated threshold value. At this point, the state of the element 532 may be deemed changed, as the charge response of the element meets or exceeds the designated threshold level.
In an embodiment, hardware resources such as shown by
In an embodiment, a comparator 530 is used to detect the change in the state of the element 532 by detecting the voltage at the node 520. The comparator is provided on a signal line 552 that extends to the multiplexer 550. The comparator 530 may correspond to a switch that remains open until a voltage is encountered that is equal to or less than a designated threshold level. When the voltage from node 520 is equal to or less than the threshold level, the comparator 530 is closed. At each instance when sampling is performed, the multiplexer 550 records the value on the signal line 552, which is provided by the comparator being either open or closed. In this way, the comparator 530 reflects the state of the element 532 at any instance when the multiplexer 550 is used to sample the element 532.
A structure such as described with an embodiment of
Table 1 describes the capabilities and requirements of a device to have a specific dynamic range, according to an embodiment of the invention.
Table-1 assumes a pixel grid size of 1024*1224, with an assumed shutter speed of 1/60 seconds. The first column of Table-1 correspond to the number of bits needed for a pixel bit value. The pixel bit value sets the amount of information that can be carried about a pixel. According to one or more embodiments described above, the state of each pixel is recorded at each instance that the pixel grid is sampled. The second column provides the number of possible values that may be determined from each pixel after processing, and equals 2n, where n is from the first column. The third column represents the pixel sampling frequency. In an embodiment, the pixel sampling frequency is limited by the pixel bit size, and is represented for a full second to provide normality in the analysis of the various values. The fifth column represents the master sampling frequency. The fourth column illustrates the dynamic range (dB) of light intensity (using light units such as LUX) that can be achieved using the sampling frequencies stated in a particular row.
With reference to Table-1, one or more embodiments enable the use of a 10-bit pixel value to have a dynamic range of light intensity that is just less than 63 dB, with a sampling frequency set by a master clock of less than 63 MHz. This dynamic range is comparable to conventional digital cameras, and requires only a 10-bit pixel value. A 16-bit pixel value may achieve a dynamic range of light intensity that is just under 99 dB, using a master sampling frequency of just over 1 GHz. However, Table-1 assumes that a single master buffer is used for the 1024*1224 size pixel grid. If the pixel grid is divided to use separate buffers, the master sampling frequency can be similarly reduced. Thus, additional buffers and hardware can compensate for the master sampling frequency.
In general, the maximum level of light intensity that can be captured on camera is about 20,000 LUX, while the darkest level of light intensity is in a range of 100 to 1000 LUX. The dynamic range of these two extremes is about 140 dB. According to embodiments described herein, and illustrated by Table-1, this dynamic range can be achieved by sampling individual pixels at 500 MHz, collecting 8388 matrices of the pixel grid in the shutter time ( 1/60 second), and using 23 bits to store each pixel value.
Although illustrative embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments. As such, many modifications and variations will be apparent to practitioners skilled in this art. Accordingly, it is intended that the scope of the invention be defined by the following claims and their equivalents. Furthermore, it is contemplated that a particular feature described either individually or as part of an embodiment can be combined with other individually described features, or parts of other embodiments, even if the other features and embodiments make no mention of the particular feature. Thus, the absence of describing combinations should not preclude the inventor from claiming rights to such combinations.
