Patent Application
20220198604
Image sensors typically used in real-time applications, e.g., automotive, surveillance, and robotic applications, typically provide raw sensor data in the form of a color filter array (CFA) which is an m×n array of pixels (where m and n are the dimensions of the sensor) in which each pixel carries information about one color channel. In the de-facto Bayer 2×2 CFA, 50% of the pixels are green, 25% are red, and 25% are blue. For consumption by a human or machine, an image processing pipeline performs a process referred to as CFA interpolation in which pixels are up-sampled from one color per location to three colors per location.
A new type of image sensor referred to as an RGB-Infrared (RGB-IR) sensor is becoming of interest for use in vision applications that need to operate safely during the day and at night, e.g., in-cabin driver monitoring systems. An RGB-IR sensor provides an infrared (IR) pixel in addition to the red, green, and blue pixels. That is, an RGB-IR sensor provides raw sensor data in a 4×4 CFA pattern in which one pixel is an IR pixel, one is a red pixel, one is a green pixel, and one is a blue pixel. Such sensor output does not map well to image system processors (ISPs) currently used in real-time applications due to the presence of the additional IR pixel plane.
Embodiments of the present disclosure relate to processing of RGB-Infrared (RGB-IR sensor data. In one aspect, a method for processing RGB-Infrared (RGB-IR) sensor data is provided that includes receiving a raw RGB-IR image, determining whether to process the raw RGB-IR image in day mode or night mode, generating, when day mode is determined, an infrared (IR) subtracted raw Bayer image from the raw RGB-IR image and processing the IR subtracted raw Bayer image in an image signal processor (ISP), and generating, when night mode is determined, an IR image from the raw RGB-IR image.
In one aspect, a system-on-a-chip (SoC) is provided that includes an image signal processor (ISP) configured to process a raw RGB-Infrared (RGB-IR) image in day mode or night mode, wherein, when configured for day mode, the ISP generates an infrared (IR) subtracted raw Bayer image from the raw RGB-IR image and, when configured for night mode, the ISP generates an IR image from the RGB-IR image, a memory configured to store software instructions implementing a control program for the ISP, wherein the control program comprises software instructions to determine whether to process the raw RGB-IR image in day mode or night mode and to configure the ISP accordingly, and at least one processor coupled to the memory to execute the software instructions.
In one aspect, a color filter array (CFA) interpolation component is provided that includes a plurality of filters, wherein, when the CFA interpolation component is operated in a first mode, a first filter and a second filter of the plurality of filters are configured to interpolate a raw Bayer image from a raw RGB-infrared (RGB-IR) image and a third filter of the plurality of filters is configured to interpolate an IR image from the raw RGB-IR image, a coefficient table configured to store coefficients for each filter of the plurality of filters, a pixel storage block configured to store raw RGB-infrared (RGB-IR) image pixels from the raw RGB-IR image for interpolation by the first filter, the second filter and third filter, an output select component coupled to the plurality of filters and configured to select outputs of the first filter and the second filter in a correct order for the raw Bayer image, and an IR subtract component configured to perform IR subtraction on the raw Bayer image using the IR image.
Specific embodiments of the disclosure are described herein in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency
While this approach allows ISPs that are not designed to process sixteen channel raw RGB-IR image data to support RGB-IR processing, the images generated from the Bayer data may have clipping and banding artifacts. Further, night mode IR illuminated images tend to have poor resolution.
The IR subtraction 206 is improved over the prior art IR subtraction 108 to include corner case artifact mitigation and improved color fidelity. To improve image quality, rather than performing simple IR subtraction in which each IR pixel value 207 is subtracted from the color pixel value in the corresponding location in the raw Bayer image 206, e.g., C_Out=C_In−IR_In, the value of the IR pixel is mixed with the value of the corresponding color pixel, e.g., C_Out=Wc*C_In+WIR*IR_in//Wc is close to 1 and WIR is close to −1 and C=R, B, or G. The weights may be programmable and each color channel may have a different weight. In addition, thresholding may be applied to help mitigate the effects of some corner cases. The use of thresholds is explained in more detail herein in reference to
In night mode 204, the general flow 212 of processing the raw RGB-IR sensor data 214 is to generate IR image data 216 based on the raw RGB-IR image data 214. The resulting IR image data is then processed by the ISP 210 to generate the final IR image data. The IR image data 216 may be generated, for example, by interpolating the raw RGB-IR image data. An example of such interpolation is described herein in reference to
The values of the other two red pixels R-3 and R-9 in the four Bayer CFA arrays 302 are interpolated from nearby red pixels in the RGB-IR image as the pixels in the corresponding locations of the RGB-IR CFA array 300 are blue. As shown in
The four blue pixels in the four Bayer CFA arrays 302 correspond to locations of the four IR pixels in the RGB-IR CFA array 300 and the values of these pixels are interpolated from diagonally neighboring blue pixels of the IR pixels in the RGB-IR image. As shown
The IR pixels IR-0, IR-2, IR-8, and IR-10 in the IR array 402 correspond to locations of the four Gr pixels Gr-0, Gr-2, Gr-8, and Gr-10 in the RBG-IR array 400. As shown in
The IR pixels IR-1, IR-3, IR-9, and IR-11 in the IR array 402 correspond to locations of the red and blue pixels R-1, B-3, B-9, and R-11 in the RBG-IR array 400. As shown in
At the other end of the spectrum, the IR plane could have much higher illumination than the color planes which can cause color artifacts due to clipping. To mitigate such issues, an IR threshold is used to prevent color inversion caused by the value of an RGFB pixel being reduced to zero or close to zero due to IR subtraction.
Referring again to
If the RGB pixel value does not exceed this threshold, then IR thresholding is considered 502. If the IR value corresponding to the RGB pixel value does not exceed an IR threshold IR_THR, then IR subtraction is performed 510 using the corresponding IR value. If the IR value corresponding to the RGB pixel value exceeds 502 an IR threshold IR_THR, then the IR value used for IR subtraction is clipped 504 to the IR_THR value and IR subtraction is performed 506 using the clipped IR value.
In some embodiments, the IR channel threshold is a static threshold. Any suitable value may be used for the static IR channel threshold and such value may be determined empirically based, for example, on the usage scenario for the RGB-IR sensor. In some embodiments, the IR channel threshold is dynamic and is computed as a percentage of the incoming R, G, or B pixel value. For example, if the percentage is 75%, R=3500, and IR=3400, then IR_THR=min(0.75*3500, 3400)=2625. Any suitable value may be used for the percentage to compute the dynamic IR channel threshold and such value may be determined empirically based, for example, on the usage scenario for the RGB-IR sensor.
There may also corner cases in which some pixels in the RGB array are saturated and the others are not. In some embodiments, to mitigate this issue, the saturation threshold decision is taken not at a pixel boundary but rather at the boundary of a 2×2 Bayer array. Thus, if any color pixel in the 2×2 Bayer array (after interpolation) is saturated, IR subtraction is not performed on any pixel in the array.
As previously mentioned here in reference to night mode 204 in
To generate IR pixels in the IR image, a gain adjustment is applied to each pixel in each 4×4 RGB-IR CFA array followed by application of an all-pass filter. There is an independent gain value for each of the sixteen phases in the 4×4 RGB-IR array. The gain values may be chosen to resolve the coarse delta between the responses of the color channels and IR channels. Because the QE ratio between the color channels and the IR channels is not constant across the full IR spectrum, applications of the gains does not resolve all the differences. The all pass filter is designed to smooth out the residual differences between the channels.
Once the histograms are computed, each histogram is summed 702 beginning with the first bin, bin 0, until 50% of the total number of pixels in the histogram is included in the sum. If the difference 704 between the bin number of the green histogram where the green sum reached the 50% pixel count and the bin number of the IR histogram where the IR sum reached the 50% pixel count is greater than 704 a threshold, then day mode is selected 706. Otherwise, night mode is selected 708. The threshold may be empirically determined, e.g., by examining the histograms for IR and green channels under various lighting conditions. In some embodiments, the day mode versus night mode decision applies to the image used to generate the histograms. In some embodiments, the day mode versus night mode decision applies to the image following the image used to generate the histograms.
In the above descriptions, the day mode processing to generate a raw Bayer image and an IR image from a raw RGB-IR image and the IR subtraction is performed before the resulting raw Bayer image is processed by an ISP to generate the RGB planes. Similarly, the night mode processing to generate the IR image from the raw RBG-IR image is performed before the IR image is processed by an ISP. In some embodiments, the day mode and night mode processing can be performed by an ISP using a programmable CFA interpolation component included in the ISP.
The raw FE component 802 operates as an image processing front end in the raw domain. The raw FE component 802 includes functionality to receive raw sensor data and perform raw pixel processing operations such as decompanding, defective pixel correction, and lens shading correction.
The spatial noise filter component 804 includes functionality to perform spatial noise filtering on the image received from the raw FE component 802.
The GLBCE component 806 includes functionality to perform global and local brightness and contrast enhancement on the image received from the spatial noise filter component 804.
The FCP component 808 includes functionality to perform de-mosaicing and color conversion on the image received from the GLBCE component 806. The image output by the FCP component 808 is stored in memory (not shown) for further processing by other hardware components, e.g., a vision processing accelerator included in a system-on-a-chip (SoC) with the ISP 800 and/or applications executing on processors in the SoC.
The FCP component 808 includes a programmable CFA interpolation component (not specifically shown) for generation of a full color image from the received image. A CFA interpolation component may also be referred to as a de-mosaicing component. The CFA interpolation component includes functionality for generating multi-plane output for any 2×2 raw CFA format including, for example, RCBC, RCCC, Bayer, and RGBC. As is explained in more detail herein, the CFA interpolation component can be programmed to generate a Bayer image and an IR image from an RGB-IR image. An example of such a CFA interpolation component is described in more detail herein in reference to
In this example, the control program also provides configuration parameters for the CFA interpolation component to configure the component to perform IR subtraction. Configuration of the CFA interpolation component to perform these functions is described herein in reference to
In night mode, a single pass through the ISP 800 is used to generate IR image data from the RGB-IR image data. The control program bypasses all components of the ISP 800 except the CFA interpolation component in the FCP 808. In addition, the control program provides configuration parameters for the CFA interpolation component to configure the component to apply gains to the RGB-IR image data and to apply an all-pass filter to the resulting image data to generate the IR image data. Configuration of the CFA interpolation component to perform these functions is described herein in reference to
The CFA interpolation component 1000 includes line memory 1010, a coefficient table 1006, a filter component 1002 including four configurable finite impulse response (FIR) filters F0-F3, a 6×6 pixel storage block 1004, an output select component 1012, an IR subtract component 1014, and an IR gain component 1016. Each of the four FIR filters F0-F3 can support a filter kernel up to a maximum size of 6×6 and each filter supports four phases. As used herein, the term “phase” refers to a unique pixel location in a CFA array. For example, an RGB-IR CFA array includes sixteen phases and a Bayer CFA array includes four phases.
The pixel storage block 1004 is configured to store a 6×6 “window” of pixels for each interpolation operation. The memory architecture of the CFA interpolation component 1000 is line based such that one incoming line of image data 1008 and the five line buffers 1010 are utilized to fill the pixel storage block 1004. The coefficient table 1006 stores the configurable coefficients for each of the sixteen filter phases. The coefficient table 1006 also include circuitry for tracking the current phase of the pixel being filtered and causing the filters to use the coefficients corresponding to the phase.
The separation of raw RGB-IR image data into raw Bayer image data and IR image data utilizes three of the four FIR filters in the filter component 1002. Two filters, e.g., F0 and F1, are used to interpolate the raw Bayer image data from the raw RGB-IR image data and one filter, e.g., F3, is used to interpolate the IR image data from the raw RGB-IR data. As previously described herein in reference to
Further, as illustrated in Table 1, the pixel values for upper left and lower right Bayer CFA arrays can be generated with the same interpolations in the same order and the pixel values for the upper right and lower left Bayer CFA arrays can be generated with the same interpolations in the same order. More specifically, as shown in Table 1 with reference to the RGB-IR CFA array 300 and the four Bayer CFA arrays 302 of
To generate the pixel values for the four Bayer CFA arrays, the coefficients for the four phases of the FIR filter F0 can be programmed by the control program via the coefficient table 1006 to perform the copy green, copy red, left diagonal blue, and copy green interpolations to generate the Gr, R, B, and Gb pixel values for the upper left and lower right Bayer CFA arrays and the coefficients for the four phases of the FIR filter F1 can be programmed by the control program via the coefficient table 1006 to perform the copy green, red diamond, right diagonal blue, and copy green interpolations to generate the Gr, R, B, and Gb pixel values for the upper right and lower left Bayer CFA arrays.
To complete generation of the four Bayer CFA arrays for each RGB-IR CFA array, the color channel outputs of the filters F0 and F1 need to be arranged in the correct order. The output select component 1012 is configurable to select among the outputs of the four filter FIR filters F0-F3 and can be configured to perform the needed ordering.
As previously described herein in reference to
Referring again to
The control program configures the output select component 1012 and the IR subtract component 1014 for operation during the first pass processing of RGB-IR image data when day mode is selected. Otherwise, these two components may be bypassed.
To generate an IR image from RGB-IR image data in support of night mode use of an RGB-IR sensor, the IR gain component 1016 and one of the filters, e.g., F0, of the filter component are used. The IR gain component 1016 is configurable to apply gains to RGB-IR pixels before the pixels are processed by the filter F0. The control program configures the IR gain component 1016 for operation when night mode is selected. Otherwise, the component may be bypassed. More specifically, the control program provides the sixteen gains to be applied to the sixteen phases in an RGB-IR array. Further, the control program programs the filter F0 via the coefficient table 1006 to operate as an all-pass filter on the output of the IR gain component 1016 to smooth out any residual differences between the channels.
If night mode is selected 1201, gains are applied 1208 to the raw RGB-IR pixels according to phase as previously described herein to generate an initial IR image and an all-pass filter is applied 1210 to the initial IR image to generate a final IR image.
The SoC 1300 includes numerous subsystems across different domains such as one dual-core 64-bit Arm® Cortex®-A72 microprocessor subsystem 1304, a microcontroller unit (MCU) island 1306, based on two dual-core Arm® Cortex®-R5F MCUs, four additional dual-core Arm® Cortex®-R5F MCUs 1312 in the main domain, two C66x floating point digital signal processors (DSPs) 1308, one C71x floating point, vector DSP 1310, that includes a deep-learning matrix multiply accelerator (MMA), and 3D graphics processing unit (GPU) 1313. The SoC 1300 further includes a memory subsystem 1314 including up to 8 MB of on-chip static random access memory (SRAM), an internal DMA engine, a general purpose memory controller (GPMC), and an external memory interface (EMIF) module (EMIF). In addition, the SoC 1300 includes a capture subsystem 1316 with two camera streaming interfaces, a vision processing accelerator (VPAC) 1302 including one or more image signal processors (ISPs), a depth and motion processing accelerator (DMPAC) 1318, and a video acceleration module 1320. The SoC 1300 also includes a display subsystem 1322, an ethernet subsystem 1324, a navigator subsystem 1326, various security accelerators 1328, support for system services 1330, and a variety of other interfaces 1332.
Software instructions implementing a configuration program as described herein may be stored in the memory subsystem 1314 (e.g., a computer readable medium) and may execute on one or more programmable processors of the SOC 1300, e.g., the DSP 1310. Further, the one or more ISPs in the VPAC 1302 may be, for example, embodiments of the ISP 800 of
While the disclosure has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope disclosed herein.
For example, embodiments are described herein in which the CFA interpolation component performs the IR subtraction. In some embodiments, the CFA interpolation component outputs the raw Bayer image and the IR image and the IR subtraction is performed in software to generate the IR subtracted raw Bayer image.
In another example, embodiments are described herein in which the IR gain block applies gains after a block of raw RGB-IR pixels is loaded into the pixel storage block of the CFA interpolation component and before the all-pass filter is applied. In some embodiments, the IR gain component may be placed at the input to the CFA interpolation component instead such that the gains are applied before pixels are loaded into the pixel storage block.
In another example, embodiments are described herein in which IR gains are applied by an IR block. In some embodiments, the gains are applied by appropriate configuring of the coefficients of the filter used to generate the final IR image.
In another example, embodiments are described herein in which generation of the IR image in night mode is a single pass operation. In some embodiments, the generation of the IR image is a two pass operation in which the first pass through the ISP generated the IR image and the second pass through the ISP improves the quality of the IR image.
In another example, embodiments are described herein in which a histogram of a green channel in an image is generated as a basis for selection between day mode and night mode. In other embodiments, a red or blue channel may be used.
In another example, embodiments are described herein in which the CFA interpolation component has four filters, each having four phases. In other embodiments, the CFA interpolation component may have other filter configurations.
In another example, embodiments are described herein in which diamond interpolation is used to interpolate red pixel values. In some embodiments, a gradient capability of the CFA interpolation component may be used to perform directional edge interpolation of red pixel values, e.g., horizontal edge interpolation in which the bottom red pixel of the diamond interpolation is not needed or vertical edge interpolation in which the right red pixel of the diamond interpolation is not needed.
It is therefore contemplated that the appended claims will cover any such modifications of the embodiments as fall within the true scope of the disclosure.
