This relates generally to imaging devices, and more particularly, to imaging devices having image sensor pixels on wafers that are stacked on other image readout/signal processing wafers.
Image sensors are commonly used in electronic devices such as cellular telephones, cameras, and computers to capture images. In a typical arrangement, an image sensor includes an array of image pixels arranged in pixel rows and pixel columns. Circuitry may be coupled to each pixel column for reading out image signals from the image pixels.
Imaging systems may implement convolutional neural networks (CNN) to perform feature extraction (i.e., to detect one or more objects, shapes, edges, or other scene information in an image). Feature extraction can be performed in a smaller region of interest (ROI) having a lower resolution than the entire pixel array. Typically, the analog pixel values in the lower resolution ROI are read out, digitized, and stored for subsequent processing for feature extraction and convolution steps.
Electronic devices such as digital cameras, computers, cellular telephones, and other electronic devices may include image sensors that gather incoming light to capture an image. The image sensors may include arrays of image pixels. The pixels in the image sensors may include photosensitive elements such as photodiodes that convert the incoming light into image signals. Image sensors may have any number of pixels (e.g., hundreds or thousands or more). A typical image sensor may, for example, have hundreds of thousands or millions of pixels (e.g., megapixels). Image sensors may include control circuitry such as circuitry for operating the image pixels and readout circuitry for reading out image signals corresponding to the electric charge generated by the photosensitive elements.
Storage and processing circuitry 18 may include one or more integrated circuits (e.g., image processing circuits, microprocessors, storage devices such as random-access memory and non-volatile memory, etc.) and may be implemented using components that are separate from camera module 12 and/or that form part of camera module 12 (e.g., circuits that form part of an integrated circuit that includes image sensors 16 or an integrated circuit within module 12 that is associated with image sensors 16). Image data that has been captured by camera module 12 may be processed and stored using processing circuitry 18 (e.g., using an image processing engine on processing circuitry 18, using an imaging mode selection engine on processing circuitry 18, etc.). Processed image data may, if desired, be provided to external equipment (e.g., a computer, external display, or other device) using wired and/or wireless communications paths coupled to processing circuitry 18.
In accordance with an embodiment, groups of pixel values in the analog domain may be processed to extract features associated with objects in a scene. The pixel information is not being digitized to a low resolution region of interest. The feature information extracted from a pixel array can be processed in multiple steps of a convolutional neural network (as an example) using this analog implementation to identify scene information for the system, which can then be used to decide whether or not to output pixel information at a higher resolution in that region of the scene.
Die stacking may be leveraged to allow the pixel array to connect to corresponding region of interest (ROI) processors to enable efficient analog domain feature extraction (e.g., to detect object features of interest and temporal changes for areas of the array that are not being read out at full resolution through the normal digital signal processing path). Extracted features may be temporarily stored in the analog domain, which can be used to check for changes in feature values over time and to detect changes in key features related to objects in the scene.
The image pixel array 302 may be formed on the top image sensor die 202. Pixel array 302 may be organized into groups sometimes referred to as “tiles” 304. Each tile 304 may, for example, include 256×256 image sensor pixels. This tile size is merely illustrative. In general, each tile 304 may have a square shape, a rectangular shape, or an irregular shape of any suitable dimension (i.e., tile 304 may include any suitable number of pixels).
Each tile 304 may correspond to a respective “region of interest” (ROI) for performing feature extraction. A separate ROI processor 330 may be formed in the analog die 204 below each tile 304. Each ROI processor 330 may include a row shifter register 332, a column shift register 336, and row control and switch matrix circuitry for selectively combining the values from multiple neighboring pixels, as represented by converging lines 336. Signals read out from each ROI processor 330 may be fed to analog processing and multiplexing circuit 340 and provided to circuits 342. Circuits 342 may include analog filters, comparators, high-speed ADC arrays, etc. Sensor control 318 may send signals to ROI controller 344, which controls how the pixels are read out via the ROI processors 330. For example, ROI controller 344 may optionally control pixel reset, pixel charge transfer, pixel row select, pixel dual conversion gain mode, a global readout path enable signal, a local readout path enable signal, switches for determining analog readout direction, ROI shutter control, etc. Circuits 330, 340, 342, and 344 may all be formed within the analog die 204.
An imaging system configured in this way may support content aware sensing. The analog readout path supports rapid scanning for shape/feature detection, non-destructive intensity thresholding, temporal events, and may also use on-board vision smart components to process shapes. The high-speed ROI readout path can also allow for digital accumulation and burst readout without impact to the normal frame readout. This content aware sensor architecture reads out different regions at varying resolutions (spatial, temporal, bit depth) based on the importance of that part of the scene. Smart sensors are used to monitor activity/events in regions of the image that are not read out at full resolution to determine when to wake up that region for higher resolution processing. The analog feature extraction supports monitoring of activity in those particular regions of interest without going into the digital domain. Since the analog feature extraction does not require processing through an ADC, a substantial amount of power can be saved.
In the example of
Each source follower drain node SF_D within the pixel cluster may also be coupled to a group of SF drain switches 430. Switch network 430 may include a SF drain power enable switch Pwr_En_SFD that selectively connects SF_D to power supply voltage Vaa, switch Hx that selectively connects SF_D to a horizontal line Voutp_H, switch Vx that selectively connects SF_D to a vertical line Voutp_V, switch Dx that selectively connects SF_D to a first diagonal line Voutp_D1, switch Ex that selectively connects SF_D to a second diagonal line Voutp_D2, etc. Switches 430 configured in this way enables the steering of current from multiple pixel source followers to allow for summing/differencing to detect shapes and edges and connection to a variable power supply.
Each pixel output line ROI_PIX_OUT(y) within the pixel cluster may also be coupled to a group of pixel output switches 410. Switch network 410 may include a first switch Global_ROIx_out_en for selectively connecting the pixel output line to a global column output bus Pix_Out_Col(y) and a second local switch Local_ROIx_Col(y) for selectively connecting the pixel output line to a local ROI serial output bus Serial_Pix_Out_ROIx that can be shared between different columns. Configured in this way, switches 410 connects each pixel output from the ROI to one of the standard global output buses for readout, to a serial readout bus to form the circuit used to detect shapes/edges, to a high speed local readout signal chain, or a variable power supply.
Machine vision applications use algorithms to find features and objects using fundamental operations that weight groups of pixels and sum them together.
The convolution operation illustrated in
The top plate of variable weighing capacitor may be a shared DCG connection that is shared among pixels from one or more rows for better area efficiency (e.g., each variable capacitor bank may be shared among 2-4 rows of pixels, among 4-8 rows of pixels, or among more than 8 pixel rows). Arranged as such, each group of variable sized capacitors can be time shared between multiple rows of pixels. In other suitable arrangements, the shared DCG connection may be shared among multiple columns of pixels (e.g., among 2-4 columns of pixels, among 4-8 columns of pixels, or among more than 8 pixel columns), a rectangular region of pixels, or other suitable groups of pixels.
The value of shared capacitor array 600-1 may be adjusted using first select bits select_wtA. The value of shared capacitor array 600-2 may be adjusted using second select bits select_wtB. The value of shared capacitor array 600-3 may be adjusted using third select bits select_wtC. Operated in this way, the select bits can be varied to control the amount of weight that is multiplied with each pixel value being read out. In other words, the variable sized capacitor may be adjusted to set the desired amount of voltage gain for the photo-generated charge. This technique fits directly into image sensor pixels with dual conversion gain (DCG) functionality without actually changing the internal structure of a DCG pixel itself. The shared DCG capacitors are also used for normal readout of high dynamic range (HDR) signals but only a single fixed sized is used for HDR readout. Connected in this way, the selected value of the weighting capacitors will directly impact the charge and voltage at the floating-diffusion node of each individual pixel (i.e., the weighting occurs within each pixel during readout).
The pixel output line of each pixel may be selectively coupled to a negative (−) input of an integrator 620 via respective switches and resistors R. In
Positive power supply voltage Vaa may be selectively applied to the (−) integrator input via an adjustable resistor Rweight_ref by optionally asserting select_ref. Asserting the select_ref switch serves to apply a reference or reset level to the value read out at integrator 620 so that a delta can be established from the actual signal level. If desired, the select_ref switch may also be selectively asserted to apply a predetermined offset voltage to the integrating amplifier 622. The circuits within box 650 and/or the integrator 620 may be formed as part of the intermediate analog feature extraction die 204 (see
At step 684, the value of all weighting capacitors may be set (e.g., by selectively asserting the select_wtX bits controlling each variable capacitor bank 600), and the DCG switches may be turned on to couple the weighting capacitors to the corresponding floating-diffusion nodes.
At step 686, the Select_out switches associated with the positive weighted pixel values (i.e., the positive weighted pixel columns) may be activated, and integrator 620 may be allowed to integrate for a fixed period to time to allow charge at its input and output to settle. At step 688, the p1 switches may be turned off, whereas the p2 switches may be turned on to effectively flip the polarity of integrator 620.
At step 690, the Select_out switches associated with the negative weighted pixel values (i.e., the negative weighted pixel columns) may be activated, and integrator 620 may be allowed to integrate for a fixed period to time to allow charge at its input and output to settle. During this time, the charge from the negative weighted columns will subtract out from the positive weighted column values (i.e., to compute a difference between the positively weighted and negatively weighted pixel values). At step 692, a final Vneuron value may be output by amplifier 622 and subsequently captured.
Although the methods of operations are described in a specific order, it should be understood that other operations may be performed in between described operations, described operations may be adjusted so that they occur at slightly different times or described operations may be distributed in a system which allows occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations are performed in a desired way.
These steps and the voltage level of the various relevant signals associated with operating the circuitry of
At time t1, the autozero switch, the p1 switches, and the row select switches may all be turned on. At time t1 or shortly after time t1, the charge on all of the in-pixel weighting capacitors may be reset by asserting clr_DCG.
At time t2, the size of all weighting capacitors may be set (e.g., by selectively asserting the sel_wtX bits) while all of the DCG switches in the selected row may be turned on to couple the weighting capacitor banks to the associated floating diffusion nodes. In the example of
At time t3, the Select_out switches of the positive weighted columns may be activated for a fixed period of time to allow the positive weighted charge to fully settle at the integrator. After the fixed time interval, the row select switches may be turned off.
At time t5, the p1 switches may be turned off, and the p2 switches may be turned on. At this time, the row select transistors may be reactivated. At time t6, the Select_out switches of the negative weighted columns may be activated for a fixed period of time (from time t6 to t7) to allow the negative weighted charge to fully settle at the integrator. At time t8, the kernel operation for that row is complete and all switches may be turned off.
Performing input current mode MAC operations using variable capacitors in this way in the analog domain for each layer of results in the neural network saves power and area by avoiding the need to move data around to the conventional digital memories. The weighting and summing operation utilize only “passive” circuit components such as capacitors and resistors that move charge around. External memory for intermediate results is not needed because the signal processing uses the pixel FD node for storage of the neuron result and the same circuit may be used for processing the next layer in the neural network.
New kernel operations cannot necessarily use previous pixel FD node values because they might be modified or attenuated by the previous DCG weighted capacitor value. Additional gain in the readout path can also help compensate for any changes at the FD node signal value from previous operations. New kernel operations can also operate on pixels signals within a local region that are transferred to an FD node and assumed to be approximately the same value. Alternatively, new kernel operations may need to wait for photo-generate charge to be developed again within the pixel or to transfer only a portion of the pixel photodiode signal to the FD for use in this operation by modifying the high voltage value that controls the transfer gate during charge transfer.
A reset voltage may be selectively applied to each pixel output line by optionally asserting the select_ref switch (e.g., by using transistor 602 to apply a reference, reset, or offset voltage to the corresponding summing capacitor). The adjustability of summing capacitors Cin in the summing readout path may optionally provide additional gain control for improved flexibility. In other words, the kernel weights may be controlled by the in-pixel variable weighting capacitors coupled to the floating diffusion nodes via the DCG switches and/or may be controlled by the ROI-level variable summing capacitors CinX interposed in the serial output paths.
The circuits within box 650′ and/or the integrator 620′ may be formed as part of the intermediate analog feature extraction die 204 (see
At step 764, the value of all weighting capacitors may be set (e.g., by selectively asserting the select_wtX bits controlling each variable capacitor bank 600), and the DCG switches may be turned on to couple the weighting capacitors to the corresponding floating-diffusion nodes.
At step 766, the Select_out switches for both the positive and negative weighted columns may be enabled, and the output voltage read out from the pixel output lines may be routed for storage at the Cin capacitors using the respective serial/local output buses.
At step 768, the autozero switch may be turned off. At step 770, the row select switches for the negatively weighted pixel values may be turned off. At step 772, the select_ref switch may be turned on to apply a reset/reference level to the value read out at integrator 620′ so that a delta can be established from the actual signal level. At step 774, the positively weighted charge may be transferred to capacitor Cint at integrator 620′.
At step 776, the p1 switches may be turned off while the Select_out switches associated with the positively weighted pixel values (i.e., the POS weighted pixel columns) are deactivated. At step 778, the p2 switches are turned on to flip the polarity of integrator 620′, and the Select_out switches associated with the negatively weighted pixel values (i.e., the NEG weighted pixel columns) are activated.
At step 780, the select_ref switch may be turned on to apply a reset/reference level to the value read out at integrator 620′ so that a delta can be established from the actual negative weighted signal level. At step 782, the negatively weighted charge may be transferred to capacitor Cint at integrator 620′. At step 784, a final Vneuron value may be output by amplifier 622 and subsequently captured.
These steps are merely illustrative and are not intended to limit the present embodiments. At least some of the existing steps may be modified or omitted; some of the steps may be performed in parallel; additional steps may be added or inserted; and the order of certain steps may be reversed or altered.
The voltage level of the various relevant signals associated with operating the circuitry of
At time t1, the autozero switch, the p1 switches, and the row select switches may all be turned on. At time t1 or shortly after time t1, the charge on all of the in-pixel weighting capacitors may be reset by asserting clr_DCG.
At time t2, the size of all weighting capacitors may be set (e.g., by selectively asserting the sel_wtX bits) while all of the DCG switches in the selected row may be turned on to couple the weighting capacitor banks to the associated floating diffusion nodes. In the example of
At time t2, the Select_out switches for both the positive and negative weighted columns may be activated to allow the positive and negative weighted charge to accumulate at the Cin capacitors. After some time, the Select_out switches for the negative pixel values may be turned off. The row select switches may subsequently be turned off.
At time t3, the select_ref switch may be enabled to apply a reset voltage, and the resulting charge may be transferred to integrating capacitor Cint.
At time t4, the p1 switches may be turned off, and the p2 switches may be turned on. At this time, the Select_out switches for the positive weighted pixel values may be deasserted. During this time, polarity of the switched capacitor integrator 620′ is flipped and charge may be allowed to settle (see time t5).
At time t6, the Select_out switches for the negative weighted pixel values may be asserted while the select_ref switch is enabled to apply a reset voltage. During this time (from t6 to t7), the resulting charge associated with the negatively weighted pixel values may be transferred to integrating capacitor Cint. At time t8, the kernel operation for that row is complete and all switches may be turned off.
At time t1, the autozero switch, the p1 switches, and the row select switches for pixels A, B, and C may be turned on. At time t1 or shortly after time t1, the charge on all of the in-pixel weighting capacitors may be reset by asserting clr_DCG.
At time t2, the size of the positive weighting capacitors may be set (e.g., by selectively asserting the sel_wtA/B/C bits) while all the DCG switches may be turned on to couple the weighting capacitor banks to the associated floating diffusion nodes. At time t2, the Select_out switches for the positive weighted columns may also be activated to allow the positive weighted charge to accumulate at the corresponding Cin capacitors. The row select switches may subsequently be turned off.
At time t3, the select_ref switch may be enabled to apply a reset voltage, and the resulting accumulated positively weighted charge may be transferred to integrating capacitor Cint.
At time t4, the select_ref switch is disabled, and the all switches associated with the positive weighted pixels may be turned off (e.g., the p1 switches may be turned off, the Select_out switches for the positive weighted columns may be deactivated), and the DCG switches may be turned off to decouple the weighting capacitors from the floating diffusion nodes.
At time t5, the clr_DCG signal may again be pulsed high to clear the charge from the positively weighted A, B, and C pixels.
At time t6, the autozero switch may again be turned on so that the negatively weighted D and E pixel values may be stored on the Cin capacitors. At this time, the row select switches for pixels D and E may be turned on, all DCG switches may be turned on to couple the weighting capacitor banks to the associated floating diffusion nodes, the size of the negative weighting capacitors may be set (e.g., by selectively asserting the sel_wtD/E bits), and the Select_out switches for the negative weighted columns may be activated to allow the negative weighted charge to accumulate at the corresponding Cin capacitors.
At time t7, the p2 switches may be turned on. During this time, polarity of the switched capacitor integrator 620′ is flipped and charge may be allowed to settle. At time t8, the select_ref switch is enabled to apply a reset voltage. During this time (from time t8 to t9), the resulting charge associated with the negatively weighted pixel values may be transferred to integrating capacitor Cint. At time t9, the kernel operation for that row is complete and all switches may be turned off.
Performing input current mode MAC operations using only variable capacitors in these ways in the analog domain for each layer of results in the neural network saves power and area by avoiding the need to move data around to the conventional digital memories. The weighting and summing operation utilize only “passive” capacitor circuit components for moving charge around. External memory for intermediate results is not needed because the signal processing uses the pixel FD node for storage of the neuron result and the same circuit may be used for processing the next layer in the neural network.
The foregoing is merely illustrative of the principles of this invention and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. The foregoing embodiments may be implemented individually or in any combination.
This application claims the benefit of provisional patent application No. 62/886,613, filed Aug. 14, 2019, which is hereby incorporated by reference herein in its entirety.
Number | Date | Country | |
---|---|---|---|
62886613 | Aug 2019 | US |