The present disclosure is generally related to image sensing and, more particularly, to processing of images and additional information sensed by image sensors.
Unless otherwise indicated herein, approaches described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.
A color image sensor is a sensor that detects or otherwise senses electromagnetic waves in visible light spectrums as color information, or image data, that represents an image. Image sensors commonly utilize a color filter array (CFA) that passes red, green or blue light to pixels of the image sensor, forming interlaced grids sensitive to red, green and blue light. Typically, chromatic pixels are utilized in image sensors to reproduce a sensed color image and, accordingly, color information of the image is subsampled by the chromatic pixels. The color information undergoes raw processing by a processor such as an image signal processor (ISP) so that the image data can be transformed into an image for display.
Sometimes additional information other than chromatic information is desired during image acquisition process by an image sensor. A conventional approach is to change the layout of the CFA to include one or more pixels that sense the desired additional information. However, it is necessary to modify the subsequent raw processing in order to accommodate the change in the layout of the CFA and the sensed additional information. Undesirably, this may require a change in the design of the ISP and additional cost associated with such change.
The following summary is illustrative only and is not intended to be limiting in any way. That is, the following summary is provided to introduce concepts, highlights, benefits and advantages of the novel and non-obvious techniques described herein. Select, not all, implementations are further described below in the detailed description. Thus, the following summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.
An objective of the present disclosure is to propose solution(s), scheme(s), technique(s), method(s) and apparatus that address the aforementioned issue associated with the conventional approach. According to the present disclosure, pixels that sense additional information may be allocated to replace some of the chromatic pixels with the lowest contribution to luminance while the basic layout of the CFA may remain intact. Additional information of one or more types may be sensed in various implementations in accordance with the present disclosure. Advantageously, the proposed solution does not require any change to the hardware of the ISP, thereby saving the additional cost associated with the conventional approach. Put differently, the proposed solution provides the ability to obtain additional information with an image sensor during image acquisition by the image sensor, but without the additional cost of an extra sensor for the additional information.
In one example implementation, a method may involve receiving, from an array of pixels of an image sensor, data sensed by the array of pixels. The array of pixels may include a plurality of imaging pixels capable of sensing image data. The array of pixels may also include one or more information pixels capable of sensing additional information. The sensed data may include the image data and the additional information. The method may also involve obtaining the additional information from the sensed data. The method may further involve processing the sensed data in a way that is used to process the image data without the additional information in the sensed data.
In another example implementation, a method may involve receiving, from an array of pixels of an image sensor, data sensed by the array of pixels. The array of pixels may include a plurality of imaging pixels capable of sensing image data. The array of pixels may also include one or more information pixels capable of sensing additional information. The sensed data may include the image data and the additional information. The image sensor may also include a Bayer filter mosaic such that each pixel of the array of pixels is aligned with a respective filter of the Bayer filter mosaic to sense light intensity in a red, green or blue wavelength region. The method may also involve obtaining the additional information from the sensed data. The method may further involve processing the sensed data in a way that is used to process the image data without the additional information in the sensed data.
In yet another example implementation, an apparatus may include a processor. The processor may be capable of receiving data sensed by an array of pixels of an image sensor. The sensed data may include image data and additional information. The processor may be capable of obtaining the additional information from the sensed data. The processor may be also capable of processing the sensed data in a way that is used to process the image data without the additional information in the sensed data.
The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of the present disclosure. The drawings illustrate implementations of the disclosure and, together with the description, serve to explain the principles of the disclosure. It is appreciable that the drawings are not necessarily in scale as some components may be shown to be out of proportion than the size in actual implementation in order to clearly illustrate the concept of the present disclosure.
As shown in
CFA 114 may include an array of color filters such as red color filters that allow electromagnetic waves 132 in the red light spectrum to pass through, green color filters that allow electromagnetic waves 134 in the green light spectrum to pass through, and blue color filters that allow electromagnetic waves 136 in the blue light spectrum to pass through. These red (R), green (G) and blue (B) color filters may, together, be interchangeably referred to as RGB color filters herein. In some implementations, CFA 114 may have RGB color filters arranged in a pattern as a Bayer filter mosaic. That is, in some implementations, in CFA 114 the proportion of red color filters, green color filters and blue color filters may be 50% green, 25% red and 25% blue. Alternatively, CFA 114 may have RGB color filters in a proportion different than that of a Bayer filter mosaic.
When CFA 114 is disposed on or otherwise aligned with the array of pixels 112, each pixel of the array of pixels 112, whether an imaging pixel or an information pixel, is respectively aligned with a red color filter, green color filter or blue color filter of CFA 114. According to the present disclosure, the location of each information pixel is selected such that each information pixel replaces an imaging pixel with a relatively low contribution to luminance with respect to an image sensed by the array of pixels 112, while maintaining the basic layout of color filters. Among the tricolors of red, green and blue, the color blue tends to contribute to luminance the least compared to the colors red and green. Thus, in some implementations, in terms of location each of the one or more information pixels in the array of pixels 112 may replace a respective imaging pixel aligned with a blue color filter of CFA 114 to sense electromagnetic waves 136 in the blue light spectrum. Moreover, in order to ensure that the basic layout can be recovered, each information pixel is surrounded by imaging pixels that are aligned to sense the color of light which would be sensed by the imaging pixel replaced by the information pixel. In cases that each information pixel replaces an imaging pixel aligned with a blue color filter of CFA 114 to sense electromagnetic waves in the blue light spectrum, each information pixel is surrounded by imaging pixels that are aligned with CFA 114 to sense electromagnetic waves 136 in the blue light spectrum. In the example shown in
Processor 120 may receive the sensed data, including image data and additional information 138, from image sensor 110. In accordance with the present disclosure, processor 120 may obtain, determine or otherwise produce a fully-sampled image 142 based on the image data as well as obtain sampled additional information 144. One advantage provided by the present disclosure is that there is no need to alter the image processing performed by processor 120 in order to obtain the fully-sampled image 142 and the sampled additional information 144. In contrast, under conventional approaches, modification to image processing is necessary due to changes in the layout of CFA to accommodate the information pixels.
In the example shown in
Scheme 200 may be utilized regardless of the type of information sensed by the information pixel(s) in an image sensor in accordance with the present disclosure. Although a Bayer filter mosaic is utilized in scheme 200, scheme 200 may also be applicable to a different filter layout.
Following the principles of scheme 200, in each of image sensor 310 and image sensor 360, each information pixel capable of sensing additional information is situated in or otherwise occupies a location of an imaging pixel that would be aligned with a blue color filter of CFA 300 or CFA 350 to sense electromagnetic waves in the blue light spectrum. In the examples shown in
Moreover, following the principles of scheme 200, each information pixel in image sensor 310 and image sensor 360 is surrounded by imaging pixels that are aligned with CFA 300 and CFA 350, respectively, to sense electromagnetic waves in the blue light spectrum. Put differently, each information pixel in image sensor 310 and image sensor 360 is respectively surrounded by at least one row of pixels in each of two opposing column-wise directions (e.g., one row above and another row below the information pixel in concern) and at least one column of pixels in each of two opposing row-wise directions (e.g., one column to the left and another column to the right of the information pixel in concern) that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
Following the principles of scheme 200, in image sensor 410, each information pixel capable of sensing additional information is situated in or otherwise occupies a location of an imaging pixel that would be aligned with a blue color filter of CFA 400 to sense electromagnetic waves in the blue light spectrum. In the example shown in
Moreover, following the principles of scheme 200, each information pixel in image sensor 410 is surrounded by imaging pixels that are aligned with CFA 400 to sense electromagnetic waves in the blue light spectrum. Put differently, each information pixel in image sensor 410 is respectively surrounded by at least one row of pixels in each of two opposing column-wise directions (e.g., one row above and another row below the information pixel in concern) and at least one column of pixels in each of two opposing row-wise directions (e.g., one column to the left and another column to the right of the information pixel in concern) that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
Following the principles of scheme 200, in image sensor 510, each information pixel capable of sensing additional information is situated in or otherwise occupies a location of an imaging pixel that would be aligned with a blue color filter of CFA 500 to sense electromagnetic waves in the blue light spectrum. In the example shown in
Moreover, following the principles of scheme 200, each information pixel in image sensor 510 is surrounded by imaging pixels that are aligned with CFA 500 to sense electromagnetic waves in the blue light spectrum. Put differently, each information pixel in image sensor 510 is respectively surrounded by at least one row of pixels in each of two opposing column-wise directions (e.g., one row above and another row below the information pixel in concern) and at least one column of pixels in each of two opposing row-wise directions (e.g., one column to the left and another column to the right of the information pixel in concern) that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
Notably, although each of the information pixels in the examples shown in
Additionally, it is noteworthy that although the additional information sensed by the one or more information pixels may be phase-difference information and/or IR band information, the scope of the present disclosure is not limited to sensing phase-difference information, IR band information or a combination thereof. In other words, in some implementations, an image sensor in accordance with the present disclosure may include one or more information pixels capable of sensing information other than phase-difference information and IR band information such as, for example and not limited to, any environmental parameters that can be detected, monitored, measured or otherwise sensed (e.g., atmospheric pressure, humidity and the like).
Referring to
In some implementations, processor 610 may receive, from the array of pixels 622 of image sensor 620, data sensed by the array of pixels 622. The sensed data may include image data and additional information. The image data may include, for example and not limited to, chroma, luma and/or hue-related data pertaining to an image. The additional information may include, for example and not limited to, phase-difference information, IR band information, information on other environmental parameters, or a combination of some or all of the aforementioned types of information. Processor 610 may store the sensed data in memory 618. Processor 610 may also obtain the additional information from the sensed data. Moreover, processor 610 may process the sensed data in a way that is used to process the image data without the additional information in the sensed data. That is, processor 610 may be able to process the sensed data, which includes both image data and additional information, as if there is only image data included in the sensed data while processor 610 is still able to obtain or otherwise extract the additional information from the sensed data. This is one advantage provided by the proposed solution of the present disclosure over conventional approaches.
In some implementations, apparatus 600 may also include an image sensor 620 communicatively coupled to processor 610. Image sensor 620 may include an array of pixels 622 and a CFA 624. Pixels of the array of pixels 622 may be arranged in rows and columns, and may include a plurality of imaging pixels and one or more information pixels. The plurality of imaging pixels may be capable of sensing the image data. The one or more information pixels may be capable of sensing the additional information. CFA 624 may include a plurality of filters, and may be aligned with the array of pixels 622 such that each pixel of the array of pixels 622 is aligned with a respective filter of CFA 624 to sense electromagnetic waves in a red, green or blue light spectrum. In some implementations, CFA 624 may include a Bayer filter mosaic.
In some implementations, when the additional information sensed by the array of pixels 622 includes phase-difference information, the one or more information pixels may include one or more first-phase pixels capable of sensing a first phase of electromagnetic waves and one or more second-phase pixels capable of sensing a second phase of electromagnetic waves different than the first phase. Each of the one or more first-phase pixels and the one or more second-phase pixels may occupy a respective location that is aligned with CFA 624 to sense the electromagnetic waves in the blue light spectrum. Each of the one or more first-phase pixels and the one or more second-phase pixels may be respectively surrounded by at least one row of pixels in each of two opposing column-wise directions and at least one column of pixels in each of two opposing row-wise directions that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
In some implementations, when the additional information sensed by the array of pixels 622 includes IR band information, the one or more information pixels may include one or more IR pixels. Each of the one or more IR pixels may occupy a respective location that is aligned with CFA 624 to sense the electromagnetic waves in the blue light spectrum. Each of the one or more IR pixels may be respectively surrounded by at least one row of pixels in each of two opposing column-wise directions and at least one column of pixels in each of two opposing row-wise directions that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
In some implementations, in obtaining the additional information from the sensed data, processor 610 may be capable of receiving the additional information from the array of pixels 622 without processing the additional information as part of the processing of the sensed data. For example, a communication channel (e.g., electrically conductive wire(s), line(s) or pattern(s)) may be connected between each information pixel and processor 610 so that processor 610 can directly receive the sensed additional information from each information pixel. As another example, processor 610 may, for each pixel of the array of pixels 622 (regardless of whether the pixel is an imaging pixel or an information pixel), compare a value of data sensed by a respective pixel with an average of values of data sensed by neighboring pixels aligned with the CFA 624 to sense electromagnetic waves in the same light spectrum as the light spectrum the respective pixel is also aligned to sense. Based on a result of the comparison, processor 610 may be able to determine whether the value of data sensed by the respective pixel is within a predefined range relative to the average of values of data sensed by its neighboring pixels that are aligned to sense electromagnetic waves of the same light spectrum. When a difference between the value of the data sensed by the respective pixel and the average of values is within the predefined range, processor 610 may consider and treat the data sensed by the respective pixel as chromatic information albeit distorted by noise. On the other hand, when the difference between the value of the data sensed by the respective pixel and the average of values is outside the predefined range, processor 610 may consider and treat the data sensed by the respective pixel as the additional information (e.g., phase difference and/or IR band information). Accordingly, processor 610 may extract the value of the data sensed by the respective pixel as a part of the additional information in response to a determination that the value of the data sensed by the respective pixel is outside the predefined range.
In some implementations, in processing the sensed data in a way that is used to process the image data without the additional information in the sensed data, processor 610 may be capable of performing a number of operations. For instance, gain circuit 612 of processor 610 may perform a gain multiplication operation on the sensed data. Detect correction circuit 614 of processor 610 may determine whether there is abnormality in the sensed data by comparing, for each pixel of the array of pixels 622, data sensed by a respective pixel of the array of pixels 622 with data sensed by one or more other pixels neighboring the respective pixel. Detect correction circuit 614 may also correct the abnormality in response to a determination that there is abnormality in data sensed by one or more pixels of the array of pixels 622. Color interpolation circuit 616 of processor 610 may interpolate the sensed data using an interpolation method used for the image data from imaging pixels of the array of pixels 622. That is, color interpolation circuit 616 may interpolate the sensed data as if there is only image data included in the sensed data without the additional information by using an interpolation method that is used on image data as with conventional image sensors, even though the sensed data also includes the additional information.
In some implementations, in obtaining the additional information from the sensed data, processor 610 may be capable of extracting the additional information from the sensed data prior to or at the defect correction stage.
In some implementations, each pixel of the array of pixels 622 may be aligned with a respective filter of CFA 624 of image sensor 620 to sense electromagnetic waves in a red, green or blue light spectrum. In such cases, in obtaining the additional information from the sensed data, processor 610 may be capable of performing, in response to the determination that there is abnormality in data sensed by the one or more pixels of the array of pixels, a number of operations for each of the one or more pixels. For instance, processor 610 may determine whether a value of data sensed by the respective pixel is within a predefined range relative to an average of values of data sensed by neighboring pixels aligned with CFA 624 to sense electromagnetic waves in a respective light spectrum which the respective pixel is also aligned to sense. Processor 610 may also extract the value of the data sensed by the respective pixel as a part of the additional information in response to a determination that the value of the data sensed by the respective pixel is outside the predefined range.
At 710, process 700 may involve processor 610 of apparatus 600 receiving, from the array of pixels 622 of image sensor 620, data sensed by the array of pixels 622. The array of pixels 622 may include a plurality of imaging pixels capable of sensing imaging data. The array of pixels 622 may also include one or more information pixels capable of sensing additional information. The sensed data may include the imaging data and the additional information. Process 700 may proceed from 710 to 720.
At 720, process 700 may involve processor 610 of apparatus 600 obtaining the additional information from the sensed data. Process 700 may proceed from 720 to 730.
At 730, process 700 may involve processor 610 of apparatus 600 processing the sensed data in a way that is used to process the imaging data without the additional information in the sensed data.
In some implementations, the additional information may include phase-difference information. The one or more information pixels may include one or more first-phase pixels, capable of sensing a first phase of electromagnetic waves, as well as one or more second-phase pixels, capable of sensing a second phase of electromagnetic waves different than the first phase. In some implementations, the array of pixels 622 may be arranged in rows and columns. The image sensor 620 may also include a CFA 624 such that each pixel of the array of pixels 622 may be aligned with a respective filter of the CFA 624 to sense electromagnetic waves in a red, green or blue light spectrum. Each of the one or more first-phase pixels and the one or more second-phase pixels may occupy a respective location that is aligned with the CFA 624 to sense the electromagnetic waves in the blue light spectrum and may be respectively surrounded by at least one row of pixels in each of two opposing column-wise directions and at least one column of pixels in each of two opposing row-wise directions that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
In some implementations, the additional information may include IR band information. The one or more information pixels may include one or more IR pixels. In some implementations, the array of pixels 622 may be arranged in rows and columns. The image sensor 620 may also include CFA 624 such that each pixel of the array of pixels 622 may be aligned with a respective filter of the CFA 624 to sense electromagnetic waves in a red, green or blue light spectrum. Each of the one or more IR pixels may occupy a respective location that is aligned with the CFA 624 to sense the electromagnetic waves in the blue light spectrum and may be respectively surrounded by at least one row of pixels in each of two opposing column-wise directions and at least one column of pixels in each of two opposing row-wise directions that include imaging pixels aligned to sense at least the electromagnetic waves in the blue light spectrum.
In some implementations, in obtaining the additional information from the sensed data, process 700 may involve processor 610 of apparatus 600 receiving the additional information from the one or more information pixels without processing the additional information as part of the processing of the sensed data.
In some implementations, in processing the sensed data in a way that is used to process the imaging data without the additional information in the sensed data, process 700 may involve processor 610 of apparatus 600 performing a number of operations. For instance, process 700 may involve processor 610 performing, at a gain stage by gain circuit 612, a gain multiplication operation on the sensed data. Process 700 may also involve processor 610 determining, at a defect correction stage by defect correction circuit 614, whether there is abnormality in the sensed data by comparing, for each pixel of the array of pixels 622, data sensed by a respective pixel of the array of pixels 622 with data sensed by one or more other pixels neighboring the respective pixel. Process 700 may further involve processor 610 correcting, at the defect correction stage by defect correction circuit 614, the abnormality in response to a determination that there is abnormality in data sensed by one or more pixels of the array of pixels 622. Process 700 may additionally involve processor 610 interpolating, at a color interpolation stage by color interpolation circuit 616, the sensed data using an interpolation method used for the image data from imaging pixels of the array of pixels 622.
In some implementations, in obtaining the additional information from the sensed data, process 700 may involve processor 610 of apparatus 600 extracting the additional information from the sensed data prior to or at the defect correction stage.
At 810, process 800 may involve processor 610 of apparatus 600 receiving, from the array of pixels 622 of an image sensor 620, data sensed by the array of pixels 622. The array of pixels 622 may include a plurality of imaging pixels capable of sensing imaging data. The array of pixels 622 may also include one or more information pixels capable of sensing additional information. The sensed data may include the imaging data and the additional information. The image sensor 620 may also include CFA 624 which may be a Bayer filter mosaic such that each pixel of the array of pixels 622 may be aligned with a respective filter of the Bayer filter mosaic to sense electromagnetic waves in a red, green or blue light spectrum. Process 800 may proceed from 810 to 820.
At 820, process 800 may involve processor 610 of apparatus 600 obtaining the additional information from the sensed data. Process 800 may proceed from 820 to 830.
At 830, process 800 may involve processor 610 of apparatus 600 processing the sensed data in a way that is used to process the imaging data without the additional information in the sensed data.
In some implementations, the additional information may include phase-difference information, IR band information, or a combination thereof.
In some implementations, in obtaining the additional information from the sensed data, process 800 may involve processor 610 of apparatus 600 receiving the additional information from the array of pixels without processing the additional information as part of the processing of the sensed data.
In some implementations, in processing the sensed data in a way that is used to process the imaging data without the additional information in the sensed data, process 800 may involve processor 610 of apparatus 600 performing a number of operations. For instance, process 800 may involve processor 610 of apparatus 600 performing, at a gain stage by gain circuit 612, a gain multiplication operation on the sensed data. Process 800 may also involve processor 610 of apparatus 600 determining, at a defect correction stage by defect correction circuit 614, whether there is abnormality in the sensed data by comparing, for each pixel of the array of pixels 622, data sensed by a respective pixel of the array of pixels 622 with data sensed by one or more other pixels neighboring the respective pixel. Process 800 may further involve processor 610 of apparatus 600 correcting, at the defect correction stage by defect correction circuit 614, the abnormality in response to a determination that there is abnormality in data sensed by one or more pixels of the array of pixels 622. Process 800 may additionally involve processor 610 of apparatus 600 interpolating, at a color interpolation stage by color interpolation circuit 616, the sensed data using an interpolation method used for the image data from imaging pixels of the array of pixels 622.
In some implementations, in obtaining the additional information from the sensed data, process 800 may involve processor 610 of apparatus 600 extracting the additional information from the sensed data prior to or at the defect correction stage.
Alternatively or additionally, in obtaining the additional information from the sensed data comprises, in response to the determination that there is abnormality in data sensed by the one or more pixels of the array of pixels 622, process 800 may involve processor 610 of apparatus 600 performing operations for each respective pixel of the one or more pixels. For instance, process 800 may involve processor 610 of apparatus 600 determining, whether a value of data sensed by the respective pixel is within a predefined range relative to an average of values of data sensed by neighboring pixels aligned with the CFA 624 to sense electromagnetic waves in a respective light spectrum which the respective pixel is also aligned to sense. In some implementations, when a difference between the value of the data sensed by the respective pixel and the average of values is within the predefined range, it may be assumed that the data sensed by the respective pixel is chromatic information albeit distorted by noise. On the other hand, when the difference between the value of the data sensed by the respective pixel and the average of values is outside the predefined range, it may be assumed that the data sensed by the respective pixel is the additional information (e.g., phase difference and/or IR band information). Process 800 may also involve processor 610 of apparatus 600 extracting the value of the data sensed by the respective pixel as a part of the additional information in response to a determination that the value of the data sensed by the respective pixel is outside the predefined range.
From the foregoing, it will be appreciated that various implementations of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various implementations disclosed herein are not intended to be limiting, with the true scope and spirit being indicated by the following claims.
The present disclosure claims the priority benefit of U.S. Patent Application No. 62/246,770, filed on 27 Oct. 2015, which is incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62246770 | Oct 2015 | US |