METHOD AND SYSTEM FOR IMAGE PROCESSING, ELECTRONIC DEVICE AND STORAGE MEDIUM

Description

TECHNICAL FIELD

The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an image processing system, electronic equipment, and a readable storage medium.

BACKGROUND

Electronic devices such as mobile phones may be provided with a camera to realize a camera function. An image sensor for receiving light may be arranged in the camera. A filter array may be provided in the image sensor. In order to improve the signal-to-noise ratio of images acquired by electronic devices such as mobile phones, an image sensor with four-in-one pixel arrangement will be used.

SUMMARY

Embodiments of the present disclosure provide a method and system for image processing, electronic device, and computer-readable storage medium.

In a first aspect, a method for image processing is provided according to some embodiments of the present disclosure, the method is for an image sensor, the image sensor comprises a pixel array, the pixel array comprises a plurality of panchromatic photosensitive pixels and a plurality of colorful photosensitive pixels, the plurality of colorful photosensitive pixels comprise a first color photosensitive pixel, a second color photosensitive pixel and a third color photosensitive pixel with different spectral responses, the plurality of colorful photosensitive pixels have a narrower spectral response than the plurality of panchromatic photosensitive pixel, and both the first color photosensitive pixel and the third color photosensitive pixels have a narrower spectral response than the second color photosensitive pixels, the pixel array comprises a plurality of minimal repeating units, each of the minimal repeating unit comprises a plurality of subunits, each of the subunit comprises at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel; the method for image processing comprising: obtaining an original image by exposing the pixel array, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels; obtaining a colorful image based on all the colorful image pixels in the subunit, and obtaining a panchromatic image based on all the panchromatic image pixels in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; and obtaining a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel, all image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.

In a second aspect, a system for image processing is provided according to some embodiments of the present disclosure, the system comprises: an image sensor, the image sensor comprises a pixel array, the pixel array comprises a plurality of panchromatic photosensitive pixels and a plurality of colorful photosensitive pixels, the plurality of colorful photosensitive pixels comprise a first color photosensitive pixel, a second color photosensitive pixel and a third color photosensitive pixel with different spectral responses, the plurality of colorful photosensitive pixels have a narrower spectral response than the plurality of panchromatic photosensitive pixel, and both the first color photosensitive pixel and the third color photosensitive pixels have a narrower spectral response than the second color photosensitive pixels, the pixel array comprises a plurality of minimal repeating units, each of the minimal repeating unit comprises a plurality of subunits, each of the subunit comprises at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel; and a processor, the processor is configured to obtain an original image by exposing the pixel array, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels; obtain a colorful image based on all the colorful image pixels in the subunit, and obtain a panchromatic image based on all the panchromatic image pixels in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; and obtain a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel, all image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.

In a third aspect, an electronic device is provided according to some embodiments of the present disclosure, the electronic device comprises a lens, a housing, and the system for image processing described above, the lens and the system for image processing are couple to the housing, the lens and the image sensor of the system for image processing are configured to form an image by cooperation.

In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided according to some embodiments of the present disclosure, when the computer instructions are executed by a processor, the processor executes the method for image processing described above.

BRIEF DESCRIPTION OF DRAWINGS

The above additional aspects and advantages of the present disclosure will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, wherein:

FIG. 1 illustrates a schematic flow diagram of an image processing method in some embodiments of the present disclosure;

FIG. 2 illustrates a schematic structural diagram of an image processing system in some embodiments of the present disclosure;

FIG. 3 illustrates a schematic diagram of a pixel array in some embodiments of the present disclosure;

FIG. 4 illustrates a schematic cross-sectional view of a photosensitive pixel in some embodiments of the present disclosure;

FIG. 5 illustrates a pixel circuit diagram of a photosensitive pixel in some embodiments of the present disclosure;

FIG. 6 to FIG. 8 illustrate schematic diagrams of the arrangement of the smallest repeating unit in the pixel array in some embodiments of the present disclosure;

FIG. 9 illustrates a schematic diagram of the principle of acquiring a color image and a panchromatic image in some embodiments of the present disclosure;

FIG. 10 illustrates a schematic flowchart of an image processing method in some embodiments of the present disclosure;

FIG. 11 illustrates a schematic diagram of the principle of color image separation in some embodiments of the present disclosure;

FIG. 12 illustrates a schematic diagram of the principle of acquiring a target pixel of a second color in some embodiments of the present disclosure;

FIG. 13 illustrates a schematic flowchart of an image processing method in some embodiments of the present disclosure;

FIG. 14 illustrates a schematic diagram of the principle of copying the boundary of the second color initial image in some embodiments of the present disclosure;

FIG. 15 illustrates a schematic flowchart of an image processing method in some embodiments of the present disclosure;

FIG. 16 illustrates a schematic diagram of the principle of obtaining the first matrix according to the first window in the initial image of the second color in some embodiments of the present disclosure;

FIG. 17 illustrates a schematic flowchart of an image processing method in some embodiments of the present disclosure;

FIG. 18 illustrates a schematic diagram of the principle of obtaining the second matrix according to the second window in the panchromatic image according to some embodiments of the present disclosure;

FIG. 19 to FIG. 20 illustrate schematic flowcharts of image processing methods in some embodiments of the present disclosure;

FIG. 21 to FIG. 22 illustrate schematic diagrams of the principle of acquiring the target pixel of the second color in some embodiments of the present disclosure;

FIG. 23 illustrates a schematic flowchart of an image processing method in some embodiments of the present disclosure;

FIG. 24 illustrates a schematic diagram of the principle of acquiring the target pixel of the second color in some embodiments of the present disclosure;

FIG. 25 illustrates a schematic flowchart of an image processing method in some embodiments of the present disclosure;

FIG. 26 illustrates a schematic diagram of the principle of processing the initial image of the first color according to the target image of the second color to obtain the target image of the first color according to some embodiments of the present disclosure;

FIG. 27 illustrates a schematic diagram of the principle of processing an initial image of a third color according to a target image of a second color to obtain a target image of a third color according to some embodiments of the present disclosure;

FIG. 28 to FIG. 29 illustrate schematic flowcharts of image processing methods in some embodiments of the present disclosure;

FIG. 30 illustrates a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 31 illustrates a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are examples, are only for explaining the embodiments of the present disclosure, and should not be construed as limiting the embodiments of the present disclosure.

Please refer to FIG. 1, FIG. 2 and FIG. 3, the present disclosure provides a method for image processing for an image sensor 10. The image sensor 10 includes a pixel array 11, and the pixel array 11 includes a plurality of panchromatic photosensitive pixels W and a plurality of colorful photosensitive pixels. The color photosensitive pixel includes a first color photosensitive pixel A, a second color photosensitive pixel B, and a third color photosensitive pixel C with different spectral responses, wherein the colorful photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixel W, and the second color photosensitive pixel has a narrower spectral response than the panchromatic photosensitive pixel W, and The photosensitive pixel A of the first color and the photosensitive pixel C of the third color have a narrower spectral response than the second color photosensitive pixel B. The pixel array 11 includes a plurality of minimal repeating units, each of which includes a plurality of subunits, and each subunit includes at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel W. The method for image processing includes:

At block 01: obtaining an original image by exposing the pixel array 11, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels;

At block 02: obtaining a colorful image based on all the colorful image pixels in the subunit, and obtaining a panchromatic image based on all the panchromatic image pixels in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; and

At block 03: obtaining a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel, all image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.

Please refer to FIG. 10, in some embodiments, the method for image processing further comprises:

At block 04: obtaining a first color initial image A, a second color initial image B, and a third color initial image C by separating the first color image pixels, the second color image pixels, and the third color image pixels in the colorful image. block 03: the obtaining a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image comprising:

At block 031: if the panchromatic image pixel in the panchromatic image W0 corresponding to the image pixel D0 to be updated in the second color initial image is not overexposed, calculating an updated pixel value of the image pixel D0 to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; and if the panchromatic image pixel in the panchromatic image W0 corresponding to the image pixel D0 to be updated in the second color initial image is overexposed, calculating an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; until the target pixel values of the second color pixels corresponding to all image pixels B in the second color initial image are obtained, obtaining the second color target image;

At block 032: obtaining the first color target image by processing the second color target image and the first color initial image, and obtaining the third color target image by processing the second color target image and the third color initial image.

Please refer to FIG. 10 and FIG. 13, in some embodiments, the image pixel of the second color initial image comprises a second color image pixel B and a null pixel N, the pixel value of the null pixel being zero. the calculating an updated pixel value of the image pixel D0 to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel comprising:

At block 0311: selecting a first window C1 centered on the image pixel D0 to be updated in the second color initial image, and selecting a second window C2 corresponding to the first window C1 in the panchromatic image, the panchromatic image pixels W0 at the center of the second window C2 correspond to the image pixel D0 to be updated and identified as a mapped panchromatic image pixel W0;

At block 0312: obtaining a first matrix I1 based on pixel information of all image pixels in the first window C1;

At block 0313: obtaining a second matrix I2 based on the pixel values of the mapped panchromatic image pixels W0, the pixel values of all panchromatic image pixels W in the second window C2, the first matrix I1, and a preset weight function F(x); and

At block 0314: obtaining the updated target pixel value of the second color pixel of the image pixel D0 to be updated based on the pixel values of the mapped panchromatic image pixels W0, the pixel values of all image pixels in the first window C1, the pixel values of all panchromatic image pixels W in the second window C2, and the second matrix I2.

Please refer to FIG. 13 and FIG. 15, in some embodiments, block 0312: obtaining a first matrix based on pixel information of all image pixels in the first window, comprising:

At block 03121: mapping the array arrangement of the image pixels in the first window C1 to the array arrangement of the first matrix I1;

At block 03122: identifying the value in the first matrix I1 corresponding to the null pixel in the first window C1 as a first value; and

At block 03123: identifying the value of the position in the first matrix I1 corresponding to the second color image pixel B in the first window as a second value.

In some embodiments, the image pixels in the first window and the second window are arranged in M*M, M representing an odd number and the first matrix being arranged in M*M.

Please refer to FIG. 13 and FIG. 17, in some embodiments, block 0313: obtaining a second matrix I2 based on the pixel values of the mapped panchromatic image pixels W0, the pixel values of all panchromatic image pixels W in the second window C2, the first matrix I1, and a preset weight function F(x), comprises:

At block 03131: mapping the matrix arrangement of image pixels in the second window C2 to the array arrangement of the second matrix I2;

At block 03132: obtaining the deviation value L1 corresponding to the position of the panchromatic image pixel W in the second matrix I2 based on the pixel value of each panchromatic image pixel W in the second window C2 and the mapped pixel value of the panchromatic image pixel W0;

At block 03133: obtaining the value of the corresponding position in the second matrix I2 based on the deviation value L1, a preset function F(x) and the value of the same position in the first matrix I1.

Please refer to FIG. 13 and FIG. 19, in some embodiments, block 0314: obtaining the updated target pixel value of the second color pixel of the image pixel to be updated based on the pixel values of the mapped panchromatic image pixels, the pixel values of all image pixels in the first window, the pixel values of all panchromatic image pixels in the second window, and the second matrix, comprises:

At block 03141, calculating the first weighted value M1 based on the first window matrix N1 formed by the pixel values of all image pixels in the first window C1 and the second matrix I2, and calculating a second weighted value M2 based on a second window matrix N2 formed by pixel values of all image pixels in the second window C2 and the second matrix I2;

At block 03142, obtaining the updated target pixel value of the second color pixel of the image pixel D0 to be updated based on the mapped pixel value of the panchromatic image pixel W0, the first weighted value M1, and the second weighted value M2.

Please refer to FIG. 10 and FIG. 20, in some embodiments, the image pixel of the first color initial image comprises a first color image pixel A and a null pixel N, the image pixel of the third color initial image comprises a third color image pixel C and a null pixel N, the value of the null N is zero; calculating an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel, comprises:

At block 0315: if the image pixel D0 to be updated is the second color image pixel B, identifying the value of the original pixel of the second color image pixel B as the value of the target pixel of the second color pixel after updating the pixel value D1 of the image pixel D0 to be updated;

At block 0316: if the position corresponding to the image pixel D0 to be updated in the first color initial image is the first color image pixel A, performing interpolation processing on the second color initial image based on the first color initial image, and obtaining the updated target pixel value D1 of the second color pixel of the image pixel D0 to be updated;

At block 0317: if the position corresponding to the image pixel D0 to be updated in the initial image of the third color is the third color image pixel C, performing interpolation processing on the second color initial image based on the third color initial image, and obtaining the updated target pixel value D1 of the second color pixel of the image pixel D1 to be updated.

Please refer to FIG. 20, in some embodiments, obtaining the first color target image by processing the second color target image and the first color initial image, and obtaining the third color target image by processing the second color target image and the third color initial image, comprises:

At block 0321: obtaining the first color target image by performing a bilateral filtering process on the first color initial image based on the second color target image, and obtaining the third color target image by performing a bilateral filtering process on the third color initial image based on the second color target image.

Please refer to FIG. 28. in some embodiments, the method for image processing further comprises:

At block 05: obtaining a processed colorful image by performing colorful image processing on the colorful image, obtaining a processed panchromatic image by performing panchromatic image processing on the panchromatic image; block 03: obtaining a fully arranged first color target image, a fully arranged second color target image and a fully arranged third color target image by processing of the color image based on the panchromatic image comprising:

At block 033: obtaining a first color target image, a second color target image and a third color target image by processing the processed colorful image based on the processed panchromatic image.

Please refer FIG. 29, in some embodiments, the method for image processing further comprises:

At block 06: obtaining a color converted target image by performing color conversion based on the first color target image, the second color target image and the third color target image.

Please refer to FIG. 1 and FIG. 2, the present disclosure provides a system 100 for image processing, the system 100 for image processing comprises an image sensor 10 and a processor 20. The image sensor 10 comprises a pixel array 11 (illustrated in FIG. 3), the pixel array 11 comprises a plurality of panchromatic photosensitive pixels W and a plurality of colorful photosensitive pixels, the plurality of colorful photosensitive pixels comprise a first color photosensitive pixel A, a second color photosensitive pixel B and a third color photosensitive pixel C with different spectral responses, the plurality of colorful photosensitive pixels W have a narrower spectral response than the plurality of panchromatic photosensitive pixel, and both the first color photosensitive pixel A and the third color photosensitive pixels C have a narrower spectral response than the second color photosensitive pixels B, the pixel array 11 comprises a plurality of minimal repeating units, each of the minimal repeating unit comprises a plurality of subunits, each of the subunit comprises at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel W. Block 01, block 02 and block 03 are all executed by the processor 20. In some embodiments, the processor 20 processor is configured to obtain an original image by exposing the pixel array, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels; obtain a colorful image based on all the colorful image pixels in the subunit, and obtain a panchromatic image based on all the panchromatic image pixels W in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; and obtain a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel A, all image pixels in the second color target image are the second color image pixels B, and all image pixels in the third color target image are the third color image pixels C.

Please refer to FIG. 2 and FIG. 10, in some embodiments, the processor 20 is further configured to obtain a first color initial image, a second color initial image, and a third color initial image by separating the first color image pixels A, the second color image pixels B, and the third color image pixels C in the colorful image; if the panchromatic image pixel W in the panchromatic image corresponding to the image pixel D0 to be updated in the second color initial image is not overexposed, calculate an updated pixel value of the image pixel D0 to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel D0 to be updated as a target pixel value of the second color pixel; and if the panchromatic image pixel in the panchromatic image W corresponding to the image pixel D0 to be updated in the second color initial image is overexposed, calculate an updated pixel value of the image pixel D0 to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identify the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; until the target pixel values of the second color pixels corresponding to all image pixels in the second color initial image are obtained, obtain the second color target image; and obtain the first color target image by processing the second color target image and the first color initial image, and obtain the third color target image by processing the second color target image and the third color initial image.

Please refer to FIG. 2, FIG. 10 and FIG. 13, in some embodiments, the processor 20 is further configured to select a first window C1 centered on the image pixel D0 to be updated in the second color initial image, and select a second window C2 corresponding to the first window C1 in the panchromatic image, the panchromatic image pixels at the center of the second window C2 correspond to the image pixel D0 to be updated and identified as a mapped panchromatic image pixel W0; obtain a first matrix I1 based on pixel information of all image pixels in the first window C1; obtain a second matrix C2 based on the pixel values of the mapped panchromatic image pixels W0, the pixel values of all panchromatic image pixels in the second window C2, the first matrix I1 and a preset weight function F(x); and obtain the updated target pixel value of the second color pixel of the image pixel to be updated based on the pixel values of the mapped panchromatic image pixels W0, the pixel values of all image pixels in the first window C1, the pixel values of all panchromatic image pixels in the second window C2, and the second matrix I2.

Please refer to FIG. 2 and FIG. 15, in some embodiments, the processor 20 is further configured to map the array arrangement of the image pixels in the first window C1 to the array arrangement of the first matrix I1; identify the value in the first matrix I1 corresponding to the null pixel in the first window C1 as a first value; and identify the value of the position in the first matrix I1 corresponding to the second color image pixel B in the first window C1 as a second value.

In some embodiments, the image pixels in the first window C1 and the second window C2 are arranged in M*M, M representing an odd number and the first matrix I1 being arranged in M*M.

Please refer to FIG. 2 and FIG. 17, in some embodiments, the processor 20 is further configured to map the matrix arrangement of image pixels in the second window C2 to the array arrangement of the second matrix I2; obtain the deviation value L1 corresponding to the position of the panchromatic image pixel W in the second matrix I2 based on the pixel value of each panchromatic image pixel W in the second window C2 and the mapped pixel value of the panchromatic image pixel W0; obtain the value of the corresponding position in the second matrix I2 based on the deviation value L1, a preset function F(x) and the value of the same position in the first matrix I1.

Please refer to FIG. 2 and FIG. 19, in some embodiments, the processor is further configured to calculate the first weighted value M1 based on the first window matrix N1 formed by the pixel values of all image pixels in the first window N1 and the second matrix I2, and calculate a second weighted value M2 based on a second window matrix N2 formed by pixel values of all image pixels in the second window C2 and the second matrix I2; obtain the updated target pixel value of the second color pixel of the image pixel D0 to be updated based on the mapped pixel value W0 of the panchromatic image pixel, the first weighted value M1, and the second weighted value M2.

Please refer to FIG. 2 and FIG. 20, in some embodiments, the processor is further configured to if the image pixel D0 to be updated is the second color image pixel B, identifying the value of the original pixel of the second color image pixel B as the value D1 of the target pixel of the second color pixel after updating the pixel value of the image pixel D0 to be updated; if the position corresponding to the image pixel D0 to be updated in the first color initial image is the first color image pixel, performing interpolation processing on the second color initial image based on the first color initial image, and obtaining the updated target pixel value of the second color pixel D1 of the image pixel D0 to be updated; and if the position corresponding to the image pixel D0 to be updated in the initial image of the third color is the third color image pixel, performing interpolation processing on the second color initial image based on the third color initial image, and obtaining the updated target pixel value D1 of the second color pixel of the image pixel D0 to be updated.

Please refer to FIG. 2 and FIG. 25, in some embodiments, the processor 20 is further configured to obtain the first color target image by performing a bilateral filtering process on the first color initial image based on the second color target image, and obtaining the third color target image by performing a bilateral filtering process on the third color initial image based on the second color target image.

Please refer to FIG. 2 and FIG. 28, in some embodiments, the processor 20 is further configured to obtain a processed colorful image by performing colorful image processing on the colorful image, obtain a processed panchromatic image by performing panchromatic image processing on the panchromatic image; and obtain a first color target image, a second color target image and a third color target image by processing the processed colorful image based on the processed panchromatic image.

Please refer to FIG. 2 and FIG. 29, in some embodiments, the processor 20 is further configured to obtain a color converted target image by performing color conversion based on the first color target image, the second color target image and the third color target image.

Please refer to FIG. 30, the present disclosure further provides an electronic device 1000. The electronic device 1000 in the embodiments of the present disclosure comprises a lens 300, a housing 200; and the system 100 for image processing of any embodiments described above, the lens 300 and the system 100 for image processing are couple to the housing 200, the lens 300 and the image sensor 10 of the system 100 for image processing are configured to form an image by cooperation.

Please refer to FIG. 31, the present disclosure further provides a non-transitory computer-readable storage medium 400 containing computer instructions, when the computer instructions are executed by a processor 60, the processor 60 executes the method for image processing described in any one of the embodiments above.

In the method for image processing and system 100 for image processing in the embodiment of the present disclosure, all colorful image pixels in the same subunit are fused into a colorful image arranged in a Bayer array; all panchromatic image pixels W in the same subunit are fused into panchromatic image, and directly use the panchromatic image and the color image to fuse, and can directly output the first color target image, the second color target image and the third color target image that contain panchromatic image information and are fully arranged. In this way, the resolution and signal-to-noise ratio of the image can be improved, and the overall photo effect can be improved.

FIG. 3 illustrates a schematic diagram of an image sensor 10 in an embodiment of the present disclosure. The image sensor 10 comprises a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.

In some embodiments, the image sensor 10 may use a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.

In some embodiments, the pixel array 11 comprises a plurality of photosensitive pixels 110 (shown in FIG. 4) arranged two-dimensionally in an array (that is, arranged in a two-dimensional matrix), and each photosensitive pixel 110 includes a photoelectric conversion element 1111 (shown in FIG. 5). Each photosensitive pixel 110 converts light into charges according to the intensity of light incident thereon.

In some embodiments, the vertical driving unit 12 comprises a shift register and an address decoder. The vertical drive unit 12 comprises readout scanning and reset scanning functions. The readout scan refers to sequentially scanning the unit photosensitive pixels 110 row by row, and reading signals from these unit photosensitive pixels 110 row by row. In some embodiments, the signal output by each photosensitive pixel 110 in the selected and scanned photosensitive pixel row is transmitted to the column processing unit 14. The reset scan is for resetting the charge, and the photo charge of the photoelectric conversion element is discarded so that accumulation of new photo charge can be started.

In some embodiments, the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing. In the CDS process, the reset level and signal level output from each photosensitive pixel 110 in the selected photosensitive pixel row are taken out, and the level difference is calculated. Thus, signals of photosensitive pixels 110 in one row are obtained. The column processing unit 14 may have an analog-to-digital (A/D) conversion function for converting analog pixel signals into digital format.

In some embodiments, the horizontal driving unit 15 comprises a shift register and an address decoder. The horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each photosensitive pixel column is sequentially processed by the column processing unit 14, and is sequentially output.

In some embodiments, the control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 12, the column processing unit 14 and the horizontal driving unit 15 to work together.

FIG. 4 is a schematic diagram of a photosensitive pixel 110 in an embodiment of the present disclosure. The photosensitive pixel 110 includes a pixel circuit 111, a filter 112, and a micro lens 113. Along the light-receiving direction of the photosensitive pixel 110, the micro lens 113, the filter 112, and the pixel circuit 111 are arranged in sequence. The micro lens 113 is used to gather light, and the filter 112 is used to allow light of a certain wavelength to pass through and filter out light of other wavelengths. The pixel circuit 111 is used to convert the received light into an electrical signal, and provide the generated electrical signal to the column processing unit 14 shown in FIG. 3.

FIG. 5 is a schematic diagram of a pixel circuit 111 of a photosensitive pixel 110 in an embodiment of the present disclosure. The pixel circuit 111 in FIG. 5 can be applied in each photosensitive pixel 110 (shown in FIG. 4) in the pixel array 11 shown in FIG. 3. The working principle of the pixel circuit 111 will be described below with reference to FIG. 3 to FIG. 5.

As illustrated in FIG. 5, the pixel circuit 111 includes a photoelectric conversion element 1111 (for example, a photodiode), an exposure control circuit (for example, a transfer transistor 1112), a reset circuit (for example, a reset transistor 1113), an amplification circuit (for example, an amplification transistor 1114) and selection circuitry (eg, selection transistor 1115). In the embodiment of the present disclosure, the transfer transistor 1112, the reset transistor 1113, the amplifying transistor 1114 and the selection transistor 1115 are, for example, MOS transistors, but are not limited thereto.

In some embodiments, the photoelectric conversion element 1111 includes a photodiode, and the anode of the photodiode is connected to the ground. Photodiodes convert received light into electric charges. The cathode of the photodiode is connected to the floating diffusion unit FD via an exposure control circuit (eg, transfer transistor 1112). The floating diffusion unit FD is connected to the gate of the amplification transistor 1114 and the source of the reset transistor 1113.

In some embodiments, the exposure control circuit is the transfer transistor 1112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 1112. When a pulse of an active level (for example, VPIX level) is transmitted to the gate of the transfer transistor 1112 through the exposure control line, the transfer transistor 1112 is turned on. The transfer transistor 1112 transfers the photoelectrically converted charge of the photodiode to the floating diffusion unit FD.

In some embodiments, the drain of the reset transistor 1113 is connected to the pixel power supply VPIX. The source of the reset transistor 113 is connected to the floating diffusion unit FD. Before the charge is transferred from the photodiode to the floating diffusion unit FD, a pulse of effective reset level is transmitted to the gate of the reset transistor 113 via the reset line, and the reset transistor 113 is turned on. The reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

In some embodiments, the gate of the amplification transistor 1114 is connected to the floating diffusion unit FD. The drain of the amplification transistor 1114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor 1113, the amplification transistor 1114 outputs a reset level through the output terminal OUT via the selection transistor 1115. After the charge of the photodiode is transferred by the transfer transistor 1112, the amplification transistor 1114 outputs a signal level through the output terminal OUT via the selection transistor 1115.

In some embodiments, the drain of the selection transistor 1115 is connected to the source of the amplification transistor 1114. The source of the selection transistor 1115 is connected to the column processing unit 14 in FIG. 3 through the output terminal OUT. When a pulse of an active level is transmitted to the gate of the selection transistor 1115 through the selection line, the selection transistor 1115 is turned on. The signal output by the amplification transistor 1114 is transmitted to the column processing unit 14 through the selection transistor 1115.

The pixel structure of the pixel circuit 111 in the embodiment of the present disclosure is not limited to the structure shown in FIG. 5. In some embodiments, the pixel circuit 111 may also have a three-transistor pixel structure, wherein the functions of the amplification transistor 1114 and the selection transistor 1115 are performed by one transistor. In some embodiments, the exposure control circuit is not limited to the way of a single transfer transistor 1112, other electronic devices or structures that have the function of controlling the conduction of the control terminal can be used as the exposure control circuit in the embodiment of the present disclosure, and the implementation of the single transfer transistor 1112 in the embodiments of the present disclosure is simple, low cost, and easy to control.

FIG. 6 to FIG. 8 are schematic diagrams of the arrangement of the light-sensitive pixels 110 (shown in FIG. 4) in the pixel array 11 (shown in FIG. 3) in some embodiments of the present disclosure. The light-sensing pixels 110 comprise two types, one is panchromatic photosensitive pixels W, and the other is colorful photosensitive pixels. The pixel array 11 can be formed by duplicating the minimum repeating unit shown in FIG. 6 to FIG. 8 multiple times on rows and columns. Each minimum repeating unit is composed of a plurality of panchromatic photosensitive pixels W and a plurality of color photosensitive pixels. Each minimal repeating unit includes multiple subunits. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of panchromatic photosensitive pixels W.

In some embodiments, FIG. 6 is a schematic diagram of arrangement of photosensitive pixels 110 (shown in FIG. 4) in the smallest repeating unit according to an embodiment of the present disclosure. Wherein, the minimum repeating unit is 16 photosensitive pixels 110 in 4 rows and 4 columns, and the sub-unit is 4 photosensitive pixels 110 in 2 rows and 2 columns. The arrangement is as follows:

W
A
W
B

A
W
B
W

W
B
W
C

B
W
C
W

W represents a panchromatic photosensitive pixel; A represents a first color photosensitive pixel among a plurality of colorful photosensitive pixels; B represents a second color photosensitive pixel among a plurality of colorful photosensitive pixels; C represents a third color photosensitive pixel among a plurality of colorful photosensitive pixels.

In some embodiments, as shown in FIG. 6, for each subunit, panchromatic photosensitive pixels W and single-color photosensitive pixels are arranged alternately.

In some embodiments, as shown in FIG. 6, the categories of subunits comprise three categories. Wherein, the first type of subunit UA comprises a plurality of panchromatic photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB comprises a plurality of panchromatic photosensitive pixels W and a plurality of second color photosensitive pixels B; the third type of sub-unit UC comprises a plurality of panchromatic photosensitive pixels W and a plurality of third color photosensitive pixels C. Each minimum repeating unit comprises four subunits, which are respectively a first-type subunit UA, two second-type subunits UB and a third-type subunit UC. Among them, a first-type subunit UA and a third-type subunit UC are arranged in the first diagonal direction D1 (such as the direction connecting the upper left corner and the lower right corner in FIG. 6), and two second-type subunits UB are arranged in the second diagonal direction D2 (for example, the direction connecting the upper right corner and the lower left corner in FIG. 6). The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

In some embodiments, the first diagonal direction D1 may also be the direction connecting the upper right corner and the lower left corner, and the second diagonal direction D2 may also be the direction connecting the upper left corner and the lower right corner. In addition, the “direction” here is not a single point, but can be understood as the concept of “straight line” indicating the arrangement, and there can be two-way pointing at both ends of the line. The explanation of the first diagonal direction D1 and the second diagonal direction D2 in FIG. 7 and FIG. 8 below is the same as here.

In some embodiments, FIG. 7 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 4) in the smallest repeating unit of another embodiment of the present disclosure. Wherein, the minimum repeating unit is 36 photosensitive pixels 110 in 6 rows and 6 columns, and the sub-unit is 9 photosensitive pixels 110 in 3 rows and 3 columns. The arrangement is as follows:

W
A
W
B
W
B

A
W
A
W
B
W

W
A
W
B
W
B

B
W
B
W
C
W

W
B
W
C
W
C

B
W
B
W
C
W

In some embodiments, as shown in FIG. 7, for each subunit, panchromatic photosensitive pixels W and single-color photosensitive pixels are arranged alternately.

In some embodiments, as shown in FIG. 7, the categories of subunits include three categories. Wherein, the first type of subunit UA comprises a plurality of panchromatic photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB comprises a plurality of panchromatic photosensitive pixels W and a plurality of second color photosensitive pixels B; The third type of sub-unit UC includes a plurality of panchromatic photosensitive pixels W and a plurality of third color photosensitive pixels C. Each minimum repeating unit comprises four subunits, which are respectively a first-type subunit UA, two second-type subunits UB and a third-type subunit UC. Wherein, one first-type subunit UA and one third-type subunit UC are arranged in the first diagonal direction D1, and two second-type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

In some embodiments, FIG. 8 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 4) in the smallest repeating unit of another embodiment of the present disclosure. Wherein, the minimum repeating unit is 64 photosensitive pixels 110 in 8 rows and 8 columns, and the sub-unit is 16 photosensitive pixels 110 in 4 rows and 4 columns. The arrangement is as follows:

W
A
W
A
W
B
W
B

A
W
A
W
B
W
B
W

W
A
W
A
W
B
W
B

A
W
A
W
B
W
B
W

W
B
W
B
W
C
W
C

B
W
B
W
C
W
C
W

W
B
W
B
W
C
W
C

B
W
B
W
C
W
C
W

In some embodiments, as shown in FIG. 8, for each subunit, panchromatic photosensitive pixels W and single-color photosensitive pixels are arranged alternately.

In some embodiments, as shown in FIG. 8, the categories of subunits comprise three categories. Wherein, the first type of subunit UA comprises a plurality of panchromatic photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB comprises a plurality of panchromatic photosensitive pixels W and a plurality of second color photosensitive pixels B; The third type of sub-unit UC comprises a plurality of panchromatic photosensitive pixels W and a plurality of third color photosensitive pixels C. Each minimum repeating unit comprises four subunits, which are respectively a first-type subunit UA, two second-type subunits UB and a third-type subunit UC. Wherein, one first-type subunit UA and one third-type subunit UC are arranged in the first diagonal direction D1, and two second-type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal and the second diagonal are perpendicular.

In some embodiments, in the minimum repeating unit shown in FIG. 6 to FIG. 8, the photosensitive pixel A of the first color can be a photosensitive pixel R of red; the second color photosensitive pixel B may be a green photosensitive pixel G; the third color photosensitive pixel C may be a blue photosensitive pixel Bu.

In some embodiments, in the minimum repeating unit shown in FIG. 6 to FIG. 8, the photosensitive pixel A of the first color can be a photosensitive pixel R of red; the second color photosensitive pixel B may be a yellow photosensitive pixel Y; the third color photosensitive pixel C may be a blue photosensitive pixel Bu.

In some embodiments, in the minimum repeating unit shown in FIG. 6 to FIG. 8, the photosensitive pixel A of the first color can be a photosensitive pixel M of magenta; The photosensitive pixel B of the second color may be a photosensitive pixel Cy of cyan; the photosensitive pixel C of the third color may be a photosensitive pixel Y of yellow.

In some embodiments, the response wavelength band of the panchromatic photosensitive pixel W may be a visible light band (for example, 400 nm-760 nm). For example, an infrared filter is arranged on the panchromatic photosensitive pixel W to filter out infrared light. In some other embodiments, the response bands of the panchromatic photosensitive pixels W are visible light bands and near-infrared bands (for example, 400 nm-1000 nm), and the photoelectric conversion elements 1111 (shown in FIG. 5) in the image sensor 10 (shown in FIG. 3) shown) to match the response band. For example, the panchromatic photosensitive pixel W may not be equipped with a filter or may be provided with a filter that allows light of all wavelengths to pass through, and the response band of the panchromatic photosensitive pixel W is determined by the response band of the photoelectric conversion element 1111, that is, the two matches. Embodiments of the present disclosure comprise but are not limited to the above-mentioned band ranges.

For the convenience of description, the following embodiments assume that the photosensitive pixels of the first color are red photosensitive pixels R; the photosensitive pixels B of the second color are green photosensitive pixels G; and the photosensitive pixels C of the third color are blue photosensitive pixels Bu.

Please refer to FIG. 9, in some embodiments, the processor 20 obtains and exposes the pixel array 11 to obtain an original image, and the original image comprises color image pixels and panchromatic image pixels W. In some embodiments, the processor 20 obtains a color image based on all the color image pixels in the same subunit, and obtains a panchromatic image based on all the panchromatic image pixels in the same subunit.

In some embodiments, please refer to FIG. 9, assuming that the original image generated after the exposure of the pixel array 11 (shown in FIG. 3) includes 16*16 image pixels, in which the colorful image pixel P1 (1,1), the colorful image pixel P1 (2, 2), panchromatic image pixel P1 (1,2) and panchromatic image pixel P1 (2,1) form a subunit U1; colorful image pixel P1 (1,3), colorful image pixel P1 (2,4), panchromatic image pixel P1 (1, 4) and panchromatic image pixel P1 (2, 3) constitute subunit U2; colorful image pixel P1 (1, 5), colorful image pixel P1 (2, 6), panchromatic image pixel P1 (1,6) and panchromatic image pixel P1 (2,5) constitute subunit U3; colorful image pixel P1 (1,7), colorful image pixel P1 (2,8), panchromatic image pixel P1 (1,8) and panchromatic image pixel P1 (2,7) constitute subunit U4, colorful image pixel P1 (1,9), colorful image pixel P1 (2,10), panchromatic image pixel P1 (1,10) and panchromatic image pixel P1 (2,11) constitutes subunit U5; colorful image pixel P1 (1,11), colorful image pixel P1 (2,12), panchromatic image pixel P1 (1,12) and panchromatic image pixel P1 (2,11) constitute subunit U6; colorful image pixel P1 (1,13), colorful image pixel P1 (2,14), panchromatic image pixel P1 (1,14) and panchromatic image pixel P1 (2,13) constitute subunit U7; colorful image pixel P1 (1,15), colorful image pixel P1 (2,16), panchromatic image pixel P1 (1,16) and panchromatic image pixel P1 (2,15) constitute subunit U8, subunit U1, subunit U2, subunit U3, subunit U4, subunit U5, subunit U6, subunit U7, and subunit U8 are located in the same row.

The processor 20 takes the average value of the pixel values of the colorful image pixel P1 (1,1) and the color image pixel P1 (2,2) in the subunit U1 as the pixel value of the fused colorful image pixel P2 (1,1), The fused colorful image pixel P2 (1,1) is located in the first row and first column of the color image; then, the processor 20 combines the colorful image pixel P1 (1,3) and the colorful image pixel P1 (2, 4) as the pixel value of the fused colorful image pixel P2 (1, 2), and the fused colorful image pixel P2 (1, 2) is located in the first row and second column of the colorful image; subsequently, the processor 20 takes the average value of the pixel values of the colorful image pixel P1 (1, 5) and the colorful image pixel P1 (2, 6) in the subunit U3 as the pixel of the fused colorful image pixel P2 (1, 3) value, the fused colorful image pixel P2 (1, 3) is located in the first row and third column of the colorful image; then, the processor 20 converts the colorful image pixel P1 (1, 7) and the colorful image pixel in the subunit U4. The mean value of the pixel value of P1 (2,8) is used as the pixel value of the fused colorful image pixel P2 (1,4), and the fused colorful image pixel P2 (1,4) is located in the first row and fourth row of the colorful image row; then, the processor 20 uses the mean value of the pixel values of the colorful image pixel P1 (1,9) and the colorful image pixel P1 (2,10) in the subunit U5 as the fused colorful image pixel P2 (1,5). The pixel value of the fused colorful image pixel P2 (1, 5) is located in the first row and fifth column of the colorful image; then, the processor 20 converts the colorful image pixel P1 (1, 11) and the colorful image pixel P1 (1, 11) in the subunit U6. The mean value of the pixel values of the image pixel P1 (2,12) is used as the pixel value of the fused colorful image pixel P2 (1,6), which is located in the first row of the colorful image Column 6; subsequently, the processor 20 uses the mean value of the pixel values of the colorful image pixel P1 (1, 13) and the colorful image pixel P1 (2, 14) in the subunit U7 as the fused colorful image pixel P2 (1, 14) 7), the fused colorful image pixel P2 (1, 7) is located in the first row and 7th column of the colorful image; then, the processor 20 converts the colorful image pixel P1 (1, 15) in the subunit U8. The average value of the pixel values of the colorful image pixel P1 (2, 16) is used as the pixel value of the fused colorful image pixel P2 (1, 8), and the fused colorful image pixel P2 (1, 8) is located in the first colorful image 1 row, 8th column. So far, the processor 20 has fused the colorful image pixels of the multiple sub-units in the first row in the original image. Subsequently, the processor 20 fuses the multiple colorful image pixels corresponding to the multiple subunits in the second row to obtain corresponding fused colorful image pixels, and then fuses the multiple colorful image pixels corresponding to the multiple subunits in the second row. The specific method of obtaining the corresponding fused colorful image pixels is the same as the specific method of merging the multiple colorful image pixels corresponding to the multiple subunits in the first row to obtain the corresponding fused colorful image pixels, and will not be repeated here. By analogy, until the processor 20 has fused the colorful image pixels of the multiple sub-units in the eighth row in the original image. In this way, all the colorful image pixels in the same subunit are fused to obtain a fused colorful image pixel, and a plurality of fused colorful image pixels are arranged to form a colorful image. The color image pixels in a colorful image are arranged in a Bayer array. Of course, the processor 20 may also average the colorful image pixels in multiple subunits at the same time to obtain multiple fused colorful image pixels, and then arrange the multiple fused colorful image pixels to generate a colorful image, which is not limited here.

Since the fused colorful image pixels in the colorful image are obtained by averaging all the colorful image pixels in the same subunit in the original image, that is, the mean value of all the colorful image pixels in the same subunit in the original image is used as the colorful image in the fused color image pixels, in this way, the obtained colorful image has a larger dynamic range than the original image, so that the dynamic range of the colorful image obtained after the colorful image is used for subsequent processing can be expanded.

In some embodiments, referring to FIG. 9, the processor 20 obtains a panchromatic image based on all the panchromatic image pixels in the same subunit. The specific implementation manner in which the processor 20 obtains a panchromatic image based on all the panchromatic image pixels in the same subunit is the same as the specific implementation manner in which the processor 20 obtains a colorful image based on all the colorful image pixels in the same subunit, and will not be repeatedly described here.

Please refer to FIG. 10, in some embodiments, the method for image processing further comprises:

At block 04: the first color image pixel A, the second color image pixel B, and the third color image pixel C in the colorful image are separated to obtain a first color initial image, a second color initial image, and a third color initial image.

Please refer to FIG. 2 and FIG. 10, block 04 may be implemented by the processor 20. In some embodiments, the processor 20 is also used to separate the first color image pixel A, the second color image pixel B, and the third color image pixel C in the colorful image to obtain the first color initial image, the second color initial image and the third color initial image.

In some embodiments, referring to FIG. 11, after the processor 20 obtains the colorful image, the processor 20 separates the first color image pixel A, the second color image pixel B, and the third color image pixel C in the colorful image, obtain the first color initial image, the second color initial image and the third color initial image. Wherein, the image pixels in the first color initial image include the first color image pixel A and the null pixel N; the image pixels in the second color initial image include the second color image pixel B and the null pixel N; the image pixels in the third color initial image include image pixels C and the null pixels N.

The following takes the acquisition of the first color initial image as an example. After the processor 20 obtains the color image, the processor 20 extracts the first color image pixel A in the colorful image and sets the extracted first color image pixel A at corresponding position of the initial image of a color. In some embodiments, the processor 20 extracts the first color image pixel A to be located in the first row and the first column of the color image, then the processor 20 sets the first color image pixel A in the first row and the first column of the first color initial image, Then the processor 20 extracts the next first color image pixel A in the colorful image, and repeats the above blocks until all the first color image pixels A in the colorful image are extracted once. The processor 20 then places a null pixel N at the position where the pixel A of the first color image is not set in the first color initial image. It should be noted that the null pixel N (NULL) is neither a panchromatic pixel nor a colorful pixel, and the position of the null pixel N in the first color initial image can be regarded as having no pixel at this position, or the pixel value of the null pixel N can be treated as zero. The specific implementation manner for the processor 20 to obtain the second color initial image and the third color initial image is the same as the specific implementation manner of obtaining the first color initial image, and will not be described one by one.

After the processor 20 obtains the first color initial image, the second color initial image and the third color initial image, please refer to FIG. 1 and FIG. 10, block 03: the obtaining a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image comprising:

Please refer to FIG. 2 and FIG. 10, block 031 and block 032 are both implemented by the processor 20. In some embodiments, the processor 20 is also configured to: if the panchromatic image pixel W in the panchromatic image corresponding to the image pixel D0 to be updated in the second color initial image is not overexposed, then calculating an updated pixel value of the image pixel D0 to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; and if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is overexposed, calculating an updated pixel value of the image pixel D0 to be updated based on the first color initial image and the second color initial image, or based on the initial image of the third color and the initial image of the second color, and use it as the target pixel of the second color pixel value; until the target pixel values of the second color pixels corresponding to all image pixels in the second color initial image are obtained, obtaining the second color target image; obtaining the first color target image by processing the second color target image and the first color initial image, and obtaining the third color target image by processing the second color target image and the third color initial image.

After the processor 20 obtains the first color initial image, the second color initial image, and the third color initial image, the processor 20 processes the second color initial image based on the panchromatic image or the first color initial image and the third color initial image. Each image pixel in the image is processed to obtain the target pixel value of the second color pixel corresponding to each image pixel, and the target pixel values of multiple second color pixels are arranged to form the second color target image.

In some embodiments, the processor 20 arbitrarily extracts any image pixel in the original image of the second color as the image pixel D0 to be updated, and the processor 20 first determines the image pixel to be updated in the panchromatic image and the original image of the second color Whether the panchromatic image pixel W0 corresponding to DO is overexposed, if the corresponding panchromatic image pixel W0 is not overexposed, then process the initial image of the second color according to the panchromatic image to obtain the second color corresponding to the image pixel D0 to be updated Color pixel target pixel value; if the corresponding panchromatic image pixel W0 is overexposed, then process the second color initial image according to the first color initial image and the third color initial image to obtain the corresponding first image pixel D0 to be updated Two-color pixel target pixel value. Subsequently, the processor 20 extracts the next image pixel in the original image of the second color as the image pixel D0 to be updated for processing, and repeats the above blocks until all the image pixels in the original image of the second color are processed. At this time, the processor 20 can obtain the target pixel value of the second color pixel corresponding to each image pixel in the second color initial image, and a plurality of second color pixel target pixel values are arranged to form the second color target image. Of course, the processor 20 can also extract the image pixels in the initial image of the second color in a certain order, for example, the processor 20 first extracts the first image pixel located in the upper left corner of the initial image of the second color for processing, and then extracts the next image pixels to be processed until the row of image pixels is processed, and then the image pixels of the next row of the second color initial image are processed, which is not limited here.

In some embodiments, the processor 20 first determines whether the panchromatic image pixel W0 corresponding to the image pixel D0 to be updated in the second color initial image is overexposed. For example, referring to FIG. 12, the image pixel D0 to be updated is located in the third row and fourth column of the second color initial image, then it is judged whether the pixel value of the panchromatic pixel W0 located in the third row and fourth column of the panchromatic image is large preset value, if the pixel value of the panchromatic pixel W0 is greater than the preset value, it can be considered that the panchromatic pixel W0 is overexposed; if the pixel value of the panchromatic pixel W0 is smaller than the preset value, it can be considered that the panchromatic pixel W0 is not overexposed, if the pixel value of the panchromatic pixel W0 is equal to the preset value, it can be considered that the panchromatic pixel W0 is overexposed or that the panchromatic pixel W0 is not overexposed. Of course, in other embodiments, it is also possible to use other methods to determine whether the panchromatic image pixel W0 corresponding to the image pixel D0 to be updated in the second-color initial image is overexposed, which will not be exemplified here.

If the panchromatic image pixel W0 corresponding to the image pixel D0 to be updated in the second color initial image in the panchromatic image is not overexposed, then calculate the updated image pixel D0 according to the panchromatic image and the pixel value of the second color initial image and as the target pixel value of the second color pixel. For example, please refer to FIG. 10 and FIG. 13, in some embodiments, the image pixels in the second-color initial image include the second-color image pixel B and a null pixel N, and the pixel value of the null pixel N is zero. Calculate the updated pixel of the image pixel D0 to be updated according to the panchromatic image and the initial image of the second color as the target pixel value of the second color pixel, comprising:

At block 0312: obtaining a first matrix I1 based on pixel information of all image pixels in the first window C1;

Please refer to FIG. 2, FIG. 10 and FIG. 13, block 0311, block 0312, block 0313 and block 0314 can all be implemented by the processor 20. In some embodiments, the processor 20 is also configured to select the first window C1 centered on the image pixel D0 to be updated in the second color initial image, and select the second window C2 corresponding to the first window C1 in the panchromatic image, the panchromatic image pixel W0 at the center of the second window C2 corresponds to the image pixel D0 to be updated and is defined as the mapped panchromatic image pixel W0; obtain the first matrix according to the pixel information of all image pixels in the first window C1 and according to the pixel value of the mapped panchromatic image pixel W0, the pixel values of all image pixels in the first window C1, the pixel values of all panchromatic image pixels W in the second window C2, and the second matrix I2, obtain the image pixel to be updated DO is the updated target pixel value of the second color pixel.

Please refer to FIG. 12, if the panchromatic image pixel W0 corresponding to the image pixel D0 to be updated in the second color initial image in the panchromatic image is not overexposed, the processor 20 selects an image to be updated in the second color initial image pixel D0 as the center of the first window C1, and select the second window C2 corresponding to the first window C1 in the panchromatic image, the panchromatic image pixel W at the center of the second window C2 corresponds to the image pixel D0 to be updated and defined as the mapped panchromatic image pixel W0. For example, the image pixel D0 to be updated is located in the third row and the fourth column in the second color initial image, then the panchromatic image pixel W corresponding to the image pixel D0 to be updated is located in the third row and the fourth column of the panchromatic image, the mapped panchromatic image pixel W0 is located in the third row and fourth column of the panchromatic image, and the image pixel D0 to be updated is located in the center of the first window C1, and the mapped panchromatic image pixel W0 is located in the center of the second window C2 Location.

In some embodiments, the first window C1 and the second window C2 are virtual computing windows, not actual structures; and the sizes of the first window C1 and the second window C2 can be changed arbitrarily according to actual needs, in some embodiments, the image pixels in the first window C1 and the second window C2 are arranged in M*M, M is an odd number, for example, M can be 3, 5, 7, 9, etc., the corresponding first window C1 and the second window C2 It can be 3*3, 5*5, 7*7 or 9*9, etc., and there is no limitation here. For the convenience of description, the following embodiments are described with the first window C1 and the second window C2 both having a size of 5*5.

In some embodiments, if the image pixel D0 to be updated is located at the edge of the initial image of the second color, that is, when the first window C1 is set with the image pixel D0 to be updated as the center, the first window C1 will be partially set outside the second color initial image. At this time, image pixels in the initial image of the second color may be set in the first window C1 centered on the image pixel D0 to be updated by copying the image boundary pixels. For example, as shown in FIG. 14, the image pixel D0 to be updated is located in the third row and the second column of the initial image of the second color. At this time, if the first window C1 is set around the image pixel D0 to be updated, the first window C1 will be partially set outside the second color initial image. By copying the image boundary pixels, that is, copying the image pixels in the first column of the image, the image pixels in the original image of the second color are set in the first window C1 centered on the image pixel D0 to be updated. Similarly, if the image pixel D0 to be updated is located at the edge of the initial image of the second color, that is, the mapped holographic image pixel W0 is also located at the edge of the panchromatic image, it is also necessary to copy the edge pixels and then set the second window The specific implementation of C2 is the same as the specific implementation of setting the first window C1 after copying the edge pixels of the initial image of the second color, and will not be repeated here.

After the processor 20 sets the first window C1 in the original image of the second color and the second window C2 in the panchromatic image, the processor 20 obtains the first matrix 11 according to the pixel information of all image pixels in the first window C1. Specifically, please refer to FIG. 13 and FIG. 15. In some embodiments, block 0312: obtaining a first matrix based on pixel information of all image pixels in the first window, comprising:

At block 03121: mapping the array arrangement of the image pixels in the first window C1 to the array arrangement of the first matrix I1;

At block 03122: identifying the value in the first matrix I1 corresponding to the null pixel in the first window C1 as a first value; and

At block 03123: identifying the value of the position in the first matrix I1 corresponding to the second color image pixel B in the first window as a second value.

Please refer to FIG. 2 and FIG. 15, block 03121, block 03122 and block 03123 can all be implemented by the processor 20. In some embodiments, the processor 20 is also used to map the array arrangement of the image pixels in the first window C1 to the array arrangement of the first matrix I1; The value of is recorded as the first value; and the value of the position corresponding to the second color image pixel B in the first window C1 in the first matrix I1 is recorded as the second value.

Please refer to FIG. 16, the processor 20 maps the array arrangement of the image pixels in the first window C1 to the array arrangement of the first matrix I1, that is, the number of rows of elements in the first matrix I1 is the same as that of the image pixels in the first window C1. The number of rows is the same, and the number of columns of elements in the first matrix 11 is the same as that of the image pixels in the first window C1, any image pixel in the first window C1 has an element corresponding to it in the first matrix I1. After obtaining the first matrix I1, the processor 20 records the value in the first matrix I1 corresponding to the null N in the first window C1 as the first value, and records the value in the first matrix I1 and the first window C1 in the first window C1 as the first value. The value of the position corresponding to the pixel B of the two-color image is recorded as the second value. In some embodiments, the first value is 0 and the second value is 1. In some embodiments, the processor 20 records the value of the position corresponding to the null N in the first matrix I1 and the first window C1 as 0, and records the value of the second color image pixel B in the first matrix I1 and the first window C1 as 0. The value of the corresponding position is recorded as 1. For example, if there are 5*5 image pixels in the first window C1, then the first matrix I1 is also a 5*5 matrix array. Among them, in the first matrix I1, the elements X11, X13, X15, X22, X24, X31, X33, X35, X42, X44, X51, X53, X55 are the positions corresponding to the null pixel N in the first window C1, and the element X12, X14, X21, X23, X25, X32, X34, X41, X43, X45, X52, X54 are positions corresponding to the null pixel N in the first window C1; The values of X31, X33, X35, X42, X44, X51, X53, and X55 are all recorded as 0, and the values of X12, X14, X21, X23, X25, X32, X34, X41, X43, X45, X52, and X54 are recorded denoted as 1, so as to obtain the first matrix I1.

Please refer to FIG. 13 and FIG. 17, in some embodiments, block 0313: according to the pixel value of the mapped panchromatic image pixel W0, the pixel values of all panchromatic image pixels W in the second window C2, the first matrix I1 and the pre-set Let the weight function F(x) obtain the second matrix I2, including:

At block 03131: mapping the matrix arrangement of image pixels in the second window C2 to the array arrangement of the second matrix I2;

Please refer to FIG. 2 and FIG. 17, block 03131, block 03132 and block 03133 can all be implemented by the processor 20. In some embodiments, the processor 20 is also configured to map the array arrangement of the image pixels in the second window C2 to the array arrangement of the second matrix I2; according to the pixel value of each panchromatic image pixel W in the second window C2 and the pixel value of the mapped panchromatic image pixel W0 to obtain the deviation value L1 of the position corresponding to the panchromatic image pixel W in the second matrix I2; and according to the deviation value L1, the preset function F(x) and the first matrix I1 For the value of the same position, the value of the corresponding position in the second matrix I2 is obtained.

After the processor 20 acquires the first matrix I1, the processor 20 maps the array arrangement of the image pixels in the second window C2 to the array arrangement of the second matrix I2. In some embodiments, the number of rows of elements in the second matrix I2 is the same as the number of rows of image pixels in the second window C2, and the number of columns of elements in the second matrix I2 is the same as the number of columns of image pixels in the second window C2, any image pixel in the second window C2 has an element corresponding to it in the second matrix I2.

The processor 20 obtains the deviation value L1 of the position corresponding to the panchromatic image pixel W in the second matrix I2 according to the pixel value of each panchromatic image pixel W in the second window C2 and the mapped pixel value of the panchromatic image pixel W0. Specifically, the deviation value L1 of the position corresponding to the panchromatic image pixel W in the second window C2 in the second matrix I2 is equal to the pixel value of the panchromatic image pixel W minus the pixel value of the mapped panchromatic image pixel W0. For example, the deviation value L1 (1,2) corresponding to row 1 and column 2 of the second matrix I2 is equal to the absolute value of the difference between the pixel values of panchromatic image pixels W (1,2) arranged in row 1 and column 2 of the panchromatic image and the pixel values of mapped panchromatic image pixels W0.

After the processor 20 obtains the deviation value L1 corresponding to all positions in the second matrix I2, the processor 20 obtains the value of the corresponding position in the second matrix I2 according to the deviation value L1, the default function F(x) and the value of the same position in the first matrix I1. Specifically, the deviation value L1 corresponding to the position to be calculated in the second matrix I2 is substituted into the default function F(x) to obtain the first result F(L1), and then the first result F(L1) is multiplied by the value of the same position in the first matrix I1 to obtain the value of the corresponding position in the second matrix I2. For example, please refer to FIG. 18. Assuming that the value of position Y12 in row 1 and column 2 of the second matrix I2 is to be calculated, the deviation value L1(1,2) corresponding to the position in row 1 and column 2 of the second matrix I2 is first substituted into the default function F(x) for calculation to obtain the first result F(L1 (1,2)). Then obtain the value of the position corresponding to Y12 in the first matrix I1, that is, obtain the value of X12 located in the first matrix I1, row 1 and column 2. The value of position Y12 in row 1 and column 2 of the second matrix I2 is equal to the first result F(L1(1,2)) multiplied by the value of X12 in the first matrix I1. It should be noted that the default function F(x) can be an exponential function, a logarithmic function or a power function, only to meet the smaller the input value, the greater the weight of the output can be, there is no limitation here.

Please refer to FIG. 2 and FIG. 19, both block 03141 and block 03142 can be implemented by the processor 20. In some embodiments, the processor 20 is also used to calculate the first weighted value M1 according to the first window matrix N1 and the second matrix I2 formed by the pixel values of all image pixels in the first window C1, and to calculate the first weighted value M1 according to the pixel values of all image pixels in the second window C2. The second window matrix N2 and the second matrix I2 formed by the pixel values of the pixels calculate the second weighted value M2; the image pixel D0 is updated to the target pixel value of the second color pixel.

After the processor 20 obtains the second matrix I2, the processor forms the first window matrix N1 according to the pixel values of all the image pixels in the first window C1, and forms the second window matrix N2 according to the pixel values of all the panchromatic image pixels W in the second window C2. It should be noted that the value of any position in the first window matrix N1 is the same as the pixel value of the image pixel at the corresponding position in the first window C1, and the value of any position in the second window matrix N2 is the same as that of the pixel values of the image pixels at the corresponding positions in the second window C2.

The processor 20 calculates a first weight M1 according to the first window matrix N1 and the second matrix I2. In some embodiments, the first weighted value M1 can be obtained by calculating the formula M1=sum(sum(H1×I2)). In some embodiments, the pixel value of each image pixel in the first window C1 is multiplied by the value at the corresponding position in the second matrix I2 to obtain multiple new pixel values, and the multiple new pixels are added to obtain the first weighted value M1. The second weighted value M2 can be obtained by calculating the formula M2=sum(sum(H2×I2)). In some embodiments, the pixel value of each panchromatic image pixel W in the second window C1 is multiplied by the value at the corresponding position in the second matrix I2 to obtain multiple new pixel values, and the multiple new pixels are added to obtain the second weighted value M2.

After the processor 20 obtains the first weighted value M1 and the second weighted value M2, the processor 20 obtains the image to be updated according to the pixel value of the mapped panchromatic image pixel W0, the first weighted value M1, the second weighted value M2, and the updated target pixel value of the second color pixel of the pixel D0. For example, the target pixel value of the second color pixel can be calculated by the formula B0′=W0′×M1/M2, wherein B0′ is the target pixel value of the second color pixel, and W0′ is the pixel value of the mapped panchromatic image pixel W0. In some embodiments, the pixel value of the mapped panchromatic image pixel W0 is multiplied by the first weighted value M1, and then divided by the second weighted value M2 to obtain the updated target pixel value of the second color pixel of the image pixel D0 to be updated.

If the panchromatic image pixel in the panchromatic image W0 corresponding to the image pixel D0 to be updated in the second color initial image is not overexposed, calculating an updated pixel value of the image pixel D0 to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel. For example, please refer to FIG. 10 and FIG. 20, in some embodiments, the image pixel of the first color initial image comprises a first color image pixel A and a null pixel N, the image pixel of the third color initial image comprises a third color image pixel C and a null pixel N, the value of the null N is zero. Calculating an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel, comprises:

Please refer to FIG. 2 and FIG. 20, block 0315, block 0316 and block 0317 can all be implemented by the processor 20. In some embodiments, the processor 20 is also configured to use the original pixel value of the image pixel B of the second color as the updated second pixel value of the image pixel D0 to be updated if the image pixel D0 to be updated is an image pixel B of the second color. Color pixel target pixel value D1; if the position corresponding to the image pixel D0 to be updated in the first color initial image is the first color image pixel, then the second color initial image is interpolated according to the first color initial image to obtain the updated second color pixel target pixel value D1 of the image pixel D0 to be updated; and if the position corresponding to the image pixel D0 to be updated in the third color initial image is a third color image pixel, then according to the third color initial image, the interpolation processing is performed on the initial image of the second color to obtain the updated target pixel value D1 of the pixel D0 of the image to be updated.

The processor 20 first determines whether the image pixel D0 to be updated is the image pixel B of the second color, and if the image pixel D0 to be updated is the image pixel B of the second color, then the original pixel value of the image pixel B of the second color is used as the image pixel B to be updated. The updated image pixel D0 is the updated target pixel value D1 of the second color pixel. In some embodiments, as shown in FIG. 21, the image pixel D0 to be updated located in the third row and second column of the initial image of the second color is the second color image pixel B, then the updated second color image pixel D0 of the image pixel to be updated is B. The target pixel value D1 of the pixel is equal to the original pixel value of the pixel B of the second color image.

If the image pixel D0 to be updated is not the second color image pixel B, that is, the image pixel D0 to be updated is a null pixel N, determine whether the image pixel at the position corresponding to the image pixel D0 to be updated in the initial image of the first color is the image pixel A of the first color, if the image pixel corresponding to the image pixel D0 to be updated in the initial image of the first color is the image pixel A of the first color, then the initial image of the second color is performed according to the initial image of the first color by processing interpolation to obtain the updated target pixel value D1 of the second color pixel of the image pixel D0 to be updated. For example, as shown in FIG. 22, the image pixel D0 to be updated located in the fifth row and fifth column of the initial image of the second color is a null pixel N, and the image pixel located in the fifth row and fifth column of the initial image of the first color is the image pixel A of the first color, the initial image of the second color is interpolated according to the initial image of the first color, so as to obtain the updated target pixel value D1 of the second color pixel of the image pixel D0 to be updated.

In some embodiments, the color difference constant theory can be used to utilize the updated target pixel value D1 of the second color pixel D0 of the image pixel D0 to be updated in the initial image of the first color. For example, please refer to FIG. 20 and FIG. 23, the first color image pixel A corresponding to the image pixel D0 to be updated in the first color initial image is defined as the mapped first color image pixel AO. The initial image of the second color is interpolated according to the initial image of the first color to obtain the updated target pixel value D1 of the second color pixel of the image pixel D0 to be updated, comprises:

At Block 03161: according to the pixel value of the image pixel in the original image of the second color and the pixel value of the image pixel in the original image of the first color, calculating the first pixel in the first direction H1 corresponding to each image pixel in the original image of the second color the difference E1 and the second difference E2 in the second direction H2;

At Block 03162: calculating the first direction difference V1 corresponding to the image pixel according to the first difference E1 corresponding to two adjacent image pixels in the first direction H1 of each image pixel in the second color initial image, according to the first direction difference E1, calculating the second difference value E2 corresponding to the image pixel corresponding to the image pixel in the second direction H2 for the second difference E2 corresponding to the image pixel of each image pixel in the second color initial image;

At Block 03163: according to the first direction difference V1 of the image pixel D0 to be updated, the second direction difference V2 of the image pixel D0 to be updated, and the first direction difference V1 and the second direction difference of the surrounding image pixels V2, calculating the first weight value g1, the second weight value g2, the third weight value g3 and the fourth weight value g4;

At Block 03164: adjacent to the first side of the image pixel D0 to be updated in the first direction H1 according to the first direction difference V1 of the image pixel D0 to be updated, the second direction difference V2 of the image pixel D0 to be updated and the first direction difference V1 of the four image pixels of D0, the first direction difference V1 of the four image pixels adjacent to the second side of the image pixel D0 in the first direction H1, and the image to be updated and the second direction difference V2 of the four image pixels adjacent to the first side of the pixel D0 in the second direction H2, and the four images adjacent to the second side of the image pixel D0 to be updated in the second direction H2, calculating the second direction difference V2, the first weight g1, the second weight g2, the third weight g3 and the fourth weight g4 of the pixel to obtain the total difference K; and

At Block 03165: acquiring the target pixel value D1 of the second color pixel corresponding to the image pixel D0 to be updated according to the mapped image pixel AO of the first color and the total difference K corresponding to the image pixel D0 to be updated.

Please refer to FIG. 2 and FIG. 23, block 03161, block 03162, block 03163, block 03164 and block 03165 can all be implemented by the processor 20. In some embodiments, the processor 20 is further configured to calculate the first difference E1 in the first direction H1 and the second difference E2 in the second direction H2 corresponding to each image pixel in the second color initial image according to the pixel value of the image pixel in the second color initial image and the pixel value of the image pixel in the first color initial image; calculate the difference V1 in the first direction corresponding to the image pixel of each image pixel in the second color based on the first difference E1 corresponding to the two adjacent image pixels in the first direction H1, calculate the second direction difference V2 corresponding to the image pixel according to the second difference E2 corresponding to two adjacent image pixels in the second direction H2 of each image pixel in the initial image of the second color; calculate the first weight value g1, the second weight value g2, the third weight value g3 and the fourth weight value g4 according to the first direction difference V1 of image pixel D0 to be updated, the second direction difference V2 of image pixel D0 to be updated, and the first direction difference V1 and the second direction difference V2 of image pixels around it; calculate to obtain the total difference K according to the first direction difference V1 of the image pixel D0 to be updated, the second direction difference V2 of the image pixel D0 to be updated, the first direction difference V1 of the four image pixels adjacent to the first side of the image pixel D0 to be updated on the first direction H1, and the first side of the four image pixels adjacent to the second side of the image pixel D0 to be updated on the first direction H1 Vector difference V1, second direction difference V2 of four image pixels adjacent to the first side of the image pixel DO to be updated on the second direction H2, second direction difference V2 of four image pixels adjacent to the second side of the image pixel D0 to be updated on the second direction H2, first weight g1, second weight g2, third weight g3 and fourth weight g4; obtain the target pixel value D1 of the second color pixel corresponding to the image pixel D0 to be updated according to the mapped first color image pixel AO and the total difference K corresponding to the image pixel D0 to be updated.

The processor 20 calculates the first direction H1 corresponding to each image pixel in the second color initial image according to the pixel value of the image pixel in the second color initial image and the pixel value of the image pixel in the first color initial image. A difference E1 and a second difference E2 in the second direction H2. It should be noted that the first direction H1 is perpendicular to the second direction H2. For convenience of illustration, in the following embodiments, the first direction H1 is the direction parallel to the columns of the image pixels, and the second direction H2 is the direction parallel to the rows of the image pixels. directions are explained. For example, assuming that the image pixel to be calculated is located in row i, column j of the original image of the second color, the corresponding first difference E1 can be calculated by the formula E1(i,j)=(B_((i,j−1))+B_((i,j+1)))/2+ custom-character (2×(A_((i,j))−A_((i,j−2))−A_((i,j+2)))/4−A_((i,j))] to obtain. Among them, B_((i,j−1)) represents the pixel value of the image pixel located in the i-th row and j−1 column of the second color initial image, and B_((i,j+1)) represents the pixel value located in the second color initial image value of the image pixel in the i-th row and j+1 column of the image, A_((i,j)) represents the pixel value of the image pixel in the i-th row and j-column of the initial image of the first color, A_((i,j−2)) represents the pixel value of the image pixel located in row i, column j−2 of the initial image of the first color, and A_((i,j+2)) represents the pixel value located in row i, column j+ of the initial image of the first color of the image pixels for the 2 columns. In some embodiments, the average value of the pixel value difference between twice the pixel value of the image pixel corresponding to the image pixel to be calculated on the initial image of the first color and the pixel value of the image pixel spaced on both sides of the first direction H1, plus the mean value of the sum of the pixel values of the image pixels on both sides of the first direction H1 is subtracted from the pixel value of the image pixel corresponding to the image pixel to be calculated on the initial image of the first color to obtain the first difference E1. The second difference E2 of the image pixel to be calculated can be calculated by the formula E2(i,j)=(B_((i−1,j))+B_((i+1,j)))/2+ custom-character (2×A_((i,j))−A_((i−2,j))−A_((i+2,j)))/4−A_((i,j))] is obtained. Among them, B_((i−1,j)) represents the pixel value of the image pixel located at row i−1, column j of the second color initial image, and B_((i+1,j)) represents the pixel value located at the second color initial image of the image pixel in the i+1th row and jth column of the image, A_((i,j)) represents the pixel value of the image pixel in the ith row and jth column of the initial image of the first color, A_((i−2,j)) represents the pixel value of the image pixel located at row i−2, column j of the initial image of the first color, and A_((i+2,j)) represents the pixel value of the image pixel located at row i+2, column j, of the initial image of the first color. In some embodiments, the average value of the pixel value difference between twice the pixel value of the image pixel corresponding to the image pixel to be calculated on the initial image of the first color and the pixel value of the image pixel spaced on both sides of the second direction H2, plus the value to be calculated. The mean value of the sum of the pixel values of the image pixels on both sides of the second direction H2 is subtracted from the pixel value of the image pixel corresponding to the image pixel to be calculated on the initial image of the first color to obtain the second difference E2. For example, please refer to FIG. 22, the image pixel to be calculated is located in the fifth row and fifth column of the original image of the second color, that is, the first difference E1 and the second difference E2 of the image pixel located in the fifth row and fifth column need to be calculated. The first difference E1 corresponding to the image pixel to be calculated can be calculated by the formula E1(5,5)=[(B_((5,4))+B_((5,6)))/2+ custom-character (2×A_((5,5))−A_((5,3))−A_((5,7)))/4−A_((5,5))] is obtained. The second difference E2 can be calculated by the formula E2(5,5)=[(B_((4,5))+B_((6,5)))/2+(2×A_((5,5))−A_((3,5))−A_((7,5)))/4−A_((5,5))] to obtain.

After the processor 20 obtains the first difference E1 and the second difference E2 corresponding to each image pixel on the second color initial image, according to the adjacent image pixels in the second color initial image in the first direction H1, calculate the first direction difference V1 corresponding to the image pixel corresponding to the first difference E1 corresponding to the two image pixels, according to the two images adjacent to each other in the second direction H2 according to the image pixels of each of the original images of the second color and the second difference E2 corresponding to the pixel calculates the second direction difference V2 corresponding to the image pixel. For example, assuming that the image pixel to be calculated is located in row i, column j of the original image of the second color, its corresponding first direction difference V1 can be calculated by the formula V1(i,j)=|E1(i,j−1)−E1(i,j+1)|, where E1(i,j−1) represents the first difference E1 corresponding to the image pixel located in row i, column j−1 on the initial image of the second color, and E1(i,j+1) represents the first difference E1 corresponding to the image pixel located at row i and column j+1 on the initial image of the second color. In some embodiments, the first direction difference V1 of the image pixel to be calculated is equal to the absolute value of the first difference E1 corresponding to two image pixels adjacent to the image pixel to be calculated in the first direction H1. The second direction difference V2 corresponding to the image pixel to be calculated can be calculated by the formula V2(i,j)=|E2(i−1,j)−E2(i+1,j)|, where E2(i+1,j) represents the second difference E2 corresponding to the image pixel located at row i+1 and column j on the initial image of the second color, and E1(i, j+1) represents the second difference E2 located at row i on the initial image of the second color and the second difference E2 corresponding to the image pixel in the j+1th column. In some embodiments, the second direction difference V2 of the image pixel to be calculated is equal to the absolute value of the difference between the second difference E2 corresponding to two adjacent image pixels of the image pixel to be calculated in the second direction H2.

After the processor 20 obtains the first direction difference V1 and the second direction difference V2 of the image pixels in the initial image of the second color, according to the first direction difference V1 of the image pixel D0 to be updated, the image pixel to be updated, the second direction difference V2 of D0, and the first direction difference V1 and the second direction difference V2 of the surrounding image pixels, calculate the first weight value g1, the second weight value g2, the third weight value g3, the first weight value g3 and the four weights g4. For example, the first weight value g1 can be calculated by the formula g1=1/(Σ_a=i−4ⁱΣ_b=j−2^j+2V2(i,j))², the image that will be located four columns to the left of the image pixel D0 to be updated and located two rows above the image pixel D0 to be updated, and within the range of two rows above the image pixel D0 to be updated and the second direction difference value V2 corresponding to the pixel is summed, and then divided by 1 by the square of the result to obtain the first weight value g1; the second weight value g2 can be calculated by the formula g2=1/(Σ_iⁱ⁺⁴Σ_b=j−2^j+2V2(i,j))², which will be located four columns to the right of the image pixel D0 to be updated and located in the to-be-updated The second direction difference V2 corresponding to the two rows above the image pixel D0 and the image pixels located in the range of the two rows above the image pixel D0 to be updated is summed, and then divided by 1 by the square of the result to obtain the second The weight value g2; the third weight value g3 can be calculated by the formula g3=1/(Σ_i−2ⁱ⁺²Σ_b=j−4^jV1(i,j))², the image pixels that will be located four rows below the image pixel D0 to be updated, two columns to the left of the image pixel D0 to be updated, and two columns to the right of the image pixel D0 to be updated corresponding first direction difference V1 is summed, and then divided by 1 by the square of the result to obtain the third weight value g3; and the fourth weight value g4 can be calculated by the formula g4=1/(Σ_i−2ⁱ⁺²Σ_b=j+44^jV1(i,j))², which will be located four rows above the image pixel D0 to be updated and located at the first direction difference V1 corresponding to the two columns on the left side of the image pixel D0 and the image pixels located in the two ranges on the right side of the image pixel D0 to be updated is summed, and then divided by 1 by the square of the result to obtain the fourth weights g4.

After the processor 20 obtains the first weight value g1, the second weight value g2, the third weight value g3 and the fourth weight value g4 of the image to be updated, the processor 20 calculates to obtain the total difference K according to the first direction difference V1 of image pixel D0 to be updated, the second direction difference V2 of image pixel D0 to be updated, the first direction difference V1 of four image pixels adjacent to the first side of image pixel D0 to be updated on the first direction H1, the first direction difference V1 of the four image pixels adjacent to the second side of the image pixel D0 to be updated in the first direction H1, the second direction difference V2 of four image pixels adjacent to the first side of the image pixel D0 to be updated in the second direction H2, the second direction difference V2 of the four image pixels adjacent to the second side of the image pixel D0 to be updated on the second direction H2, the first weight g1, the second weight g2, the third weight g3 and the fourth weight g4. For example, the first weight matrix S1 is formed according to the second difference E2 of the image pixel D0 to be updated and the 4 adjacent image pixels below it, and according to the image pixel D0 to be updated and the 4 adjacent images above it, the second difference E2 of the pixels is arranged to form the second weight matrix S2, and the third weight matrix S3 is formed according to the first difference E1 of the image pixel D0 to be updated and the four adjacent image pixels on the left side thereof, and the third weight matrix S3 is formed according to the image to be updated, the first difference E1 of the pixel D0 and its adjacent four image pixels on the right is arranged to form the fourth weight matrix S4, and the total difference K can be calculated by the formula K=(g1×f×S1+g2×f×S2+g3×f′×S3+g4×f′×S4)/(g1+g+g3+g4), where f represents the preset matrix, f′ represents the transpose of the preset matrix, and in some embodiments, the preset Matrix f=[11111]/5. For example, referring to FIG. 22, assuming that the image pixel D0 to be updated is located in row 5 and column 5 of the initial image of the second color, then the first weight matrix

$S 1 = [\begin{matrix} E 2 (1, 5) \\ E 2 (2, 5) \\ E 2 (3, 5) \\ E 2 (4, 5) \\ E 2 (5, 5) \end{matrix}];$

- the second weight matrix

$S 2 = [\begin{matrix} E 2 (5, 5) \\ E 2 (6, 5) \\ E 2 (7, 5) \\ E 2 (8, 5) \\ E 2 (9, 5) \end{matrix}];$

the third weight matrix

- S3=E1(5,1) E1(5,2) E1(5,3) E1(5,4) E1(5,5); the fourth weight matrix
- S4=E1(5,5) E1(5,6) E1(5,7) E1(5,8) E1(5,9); the total difference

$K = (g 1 \times \frac{[1 1 1 1 1]}{5} \times [\begin{matrix} E 2 (1, 5) \\ E 2 (2, 5) \\ E 2 (3, 5) \\ E 2 (4, 5) \\ E 2 (5, 5) \end{matrix}] + g 2 \times \frac{[1 1 1 1 1]}{5} \times [\begin{matrix} E 2 (5, 5) \\ E 2 (6, 5) \\ E 2 (7, 5) \\ E 2 (8, 5) \\ E 2 (9, 5) \end{matrix}] + g 3 \times \frac{1}{5} [\begin{matrix} 1 \\ 1 \\ 1 \\ 1 \\ 1 \end{matrix}] \times [⁠ \begin{matrix} E 1 (5, 1) & E 1 (5, 2) & E 1 (5, 3) & E 1 (5, 4) & E 1 (5, 5) \end{matrix}] + g 4 \times \frac{1}{5} [\begin{matrix} 1 \\ 1 \\ 1 \\ 1 \\ 1 \end{matrix}] \times [⁠ \begin{matrix} E 1 (5, 5) & E 1 (5, 6) & E 1 (5, 7) & E 1 (5, 8) & E 1 (5, 9) \end{matrix}]) \div (g 1 + g 2 + g 3 + g 4) .$

After the processor 20 obtains the total difference K, according to the mapped first color image pixel AO and the total difference K corresponding to the image pixel D0 to be updated, obtain the second color pixel target corresponding to the image pixel D0 to be updated Pixel value D1. For example, the target pixel value D1 of the second color pixel corresponding to the image pixel D0 to be updated is equal to the sum of the mapped first color image pixel AO and the total difference K corresponding to the image pixel D0 to be updated. Of course, in some embodiments, other interpolation methods may also be used to obtain the target pixel value D1 of the pixel of the second color corresponding to the image pixel D0 to be processed, and no more examples are given here.

If the image pixel D0 to be updated is not the second color image pixel B, that is, the image pixel D0 to be updated is a null pixel N, and the image pixel at the position corresponding to the image pixel D0 to be updated in the initial image of the third color is, when the third image pixel is C, the initial image of the second color is interpolated according to the initial image of the third color to obtain the updated target pixel value D1 of the second color pixel D0 of the image pixel D0 to be updated. For example, as shown in FIG. 24, the image pixel D0 to be updated located in the second row and the second column of the initial image of the second color is a null pixel N, and the image pixel located in the second row and the second column of the third color initial image is the image pixel C of the third color, the initial image of the second color is interpolated according to the initial image of the third color, so as to obtain the updated target pixel value D1 of the pixel D0 of the second color of the image pixel D0 to be updated. The specific implementation method of performing interpolation processing on the second color initial image based on the third color initial image is the same as the specific implementation method of performing interpolation processing on the second color initial image based on the first color initial image in the above embodiment, and will not be repeated here repeat.

After the processor 20 obtains the target pixel values of the second color pixels corresponding to all the second color image pixels in the second color initial image, a plurality of second color pixel target pixel values are arranged to form the second color target image. The position where the target pixel value of the second color pixel is set in the second color target image is the same as the position where the corresponding image pixel is set in the second color initial image.

Please refer to FIG. 25, after the processor 20 obtains the second color target image, in some embodiments, processes the second color target image and the first color initial image to obtain the first color target image, and processes the second color target image and a third color initial image to obtain the third color target image, comprises:

At block 0321: perform bilateral filtering processing on the first color initial image according to the second color target image to obtain a first color target image, and perform bilateral filtering processing on a third color initial image according to the second color target image to obtain a third color target image.

Please refer to FIG. 2 and FIG. 25, block 0321 can also be executed by the processor 20, that is, the processor 20 is also used to perform bilateral filtering on the initial image of the first color according to the target image of the second color to obtain the target image of the first color, according to the target image of the second color performs bilateral filtering on the initial image of the third color to obtain the target image of the third color.

In the following, description will be made by taking the second color target image as an example to perform bilateral filter processing on the first color initial image to obtain the first color target image. In some embodiments, referring to FIG. 26, the joint bilateral filtering algorithm is

$J_{p} = \frac{1}{k_{p}} \sum_{q \in Ω} I_{q} f ( p - q ) g ( I_{p}^{'} - I_{q}^{'} ),$

wherein, k_p=Σ_q∈Ωf(∥p−q∥)g(∥I_p′−I_q′∥), J_pis the output pixel value, k_pis the sum of weights, Ω is the filter window, p is the coordinate of the pixel to be filtered in the initial image of the first color, and q is the coordinate of the pixel in the filter window in the initial image of the first color, I_qis the pixel value corresponding to point q, I_p′ is the pixel value corresponding to the pixel to be filtered in the target image of the second color, I_q′ is the pixel value corresponding to point q in the target image of the second color, f and g are the weight distribution function, the weight distribution function includes a Gaussian function.

In some embodiments, the joint bilateral filtering algorithm determines the first distance weight f(∥p−q∥) by the difference between the coordinates of the pixel point p to be filtered and the coordinates of a pixel point q in the filtering window, the example p in the FIGURE The coordinate difference between the point and the q point can be 2, and the second distance weight g(∥I_p′−I_q′∥), according to the first distance weight of each pixel in the filter window, the second distance weight, the pixel value I_qcorresponding to point q in the initial image of the first color, and the weight sum k_pto determine the output pixel value J_p.

In some embodiments, in the initial image of the first color, the position of the pixel A of the first color image is not set, that is, a null pixel N, whose pixel value is 0. The second color intermediate image set by the output pixel value J_pcorresponds to the pixel point p to be filtered, and after one output is completed, the filter window moves to the next image pixel position until all image pixels in the first color initial image are completely filtered, so that the target image of the first color including only the pixel A of the first color image is obtained. Please refer to FIG. 27, the specific method for performing bilateral filtering processing on the third color initial image by the second color target image to obtain the third color target image is the same as performing bilateral filtering processing on the first color initial image by the second color target image to obtain the second color target image. The specific method for the initial image of one color is the same, and will not be repeated here.

The image processing system 100 in the embodiment of the present disclosure fuses all the color image pixels in the same subunit into a colorful image arranged in a Bayer array; all the panchromatic image pixels in the same subunit are fused into a panchromatic image, and the panchromatic image and the colorful image are directly used for fusion, and the first color target image and the second color target image that contains panchromatic image information and are fully arranged can be directly output. In this way, the resolution and signal-to-noise ratio of the image can be improved, and the overall photo effect can be improved.

Please refer to FIG. 28, in some embodiments, the method for image processing further comprises:

At block 033: obtain a first color target image, a second color target image and a third color target image by processing the processed colorful image based on the processed panchromatic image.

Please refer to FIG. 2 and FIG. 28, both block 05 and block 033 can be executed by the processor 20, that is, the processor 20 is also used to perform color image processing on the color image to obtain the processed color image, and perform panchromatic image processing on the panchromatic image to obtain a processed panchromatic image; and process the processed color image according to the processed panchromatic image to obtain a first color target image, a second color target image and a third color target image.

After the processor 20 obtains the colorful image and the panchromatic image, the processor 20 performs colorful image processing on the colorful image to obtain a processed colorful image, and performs panchromatic image processing on the panchromatic image to obtain a processed panchromatic image. It should be noted that, in some embodiments, the colorful image processing includes at least one of dead point compensation processing, vignetting compensation processing, and white balance processing; the panchromatic image processing includes dead point compensation processing.

After obtaining the processed panchromatic image and the processed color image, the processor 20 processes the processed color image according to the processed panchromatic image to obtain the first color target image, the second color target image, the second color target image and the third color target image. The specific implementation manner of processing the processed color image by the processed panchromatic image is the same as the specific implementation manner of processing the color image by the panchromatic image in the above embodiment, and will not be repeated here.

Since the color image is processed by color image processing and the panchromatic image is processed by panchromatic image, its image quality can be improved. At this time, the processed color image is processed according to the processed panchromatic image to obtain the first color target image, the second color target image and the third color target image can improve the image quality of the first color target image, the second color target image and the third color target image.

Please refer to FIG. 29, in some embodiments, the method for image processing further comprises:

At block 06: obtaining a color converted target image by performing color conversion based on the first color target image, the second color target image and the third color target image.

Please refer to FIG. 2 and FIG. 29, block 06 can also be executed by the processor 20, that is, the processor 20 is also used to perform color conversion according to the first color target image, the second color target image and the third color target image to obtain the converted color target image.

The processor 20 performs color conversion processing after obtaining the first color target image, the second color target image and the third color target image, so as to obtain a color converted target image. Color conversion processing is to convert an image from one color space (such as RGB color space) to another color space (such as YUV color space) so as to have a wider range of application scenarios or a more efficient transmission format. In a specific embodiment, the block of color conversion processing can be to convert the R, G and B channel pixel values of all pixel values in the image to obtain the Y, U and V channel pixel values by the following formula: (1) Y=0.30 R+0.59 G+0.11 B; (2) U=0.493 (B−Y); (3) V=0.877 (R−Y); thus converting the image from the RGB color space to the YUV color space. Since the luminance signal Y and the chrominance signals U and V in the YUV color space are separated, and the human eye is more sensitive to luminance than chromaticity, this application uses the first color target image, the second color target image and the third color target image to convert color to obtain the color conversion and then transmitted to the image processor (not shown) for subsequent processing, which can reduce the amount of information in the image without affecting the viewing effect of the image, thereby improving the transmission efficiency of the image.

Please refer to FIG. 30, the present disclosure also provides an electronic device 1000. The electronic device 1000 in the embodiment of the present disclosure comprises a lens 300, a housing 200 and the system 100 for image processing in any one of the above embodiments. The lens 300 and the system 100 for image processing are combined with the casing 200. The lens 300 cooperates with the image sensor 10 of the system 100 for image processing to form an image.

The electronic device 1000 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, smart glasses, a smart helmet), a drone, a head-mounted display device, etc., which is not limited here.

The electronic device 1000 in the embodiment of the present disclosure fuses all the colorful image pixels in the same subunit into a colorful image arranged in a Bayer array in the system 100 for image processing; fuses all the panchromatic image pixels in the same subunit into a panchromatic image, and directly use the panchromatic image and the colorful image to fuse, and can directly output the first color target image, the second color target image and the third color target image that contain panchromatic image information and are fully arranged. In this way, the resolution and signal-to-noise ratio of the image can be improved, and the overall photo effect can be improved.

Please refer to FIG. 31, the present disclosure also provides a non-transitory computer-readable storage medium 400 containing computer instructions. When the computer instructions are executed by the processor 60, the processor 60 is made to execute the method for image processing in any one of the above-mentioned embodiments.

For example, please refer to FIG. 1 and FIG. 31, when the computer instructions are executed by the processor 60, the processor 60 is made to perform the following blocks:

At block 01: obtain the original image by exposing the pixel array 11, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels;

At block 02: obtain a color image based on all the color image pixels in the same subunit, and obtain a panchromatic image based on all the panchromatic image pixels in the same subunit, and the image pixels in the colorful image are arranged in a Bayer array; and

At block 03: process the colorful image according to the panchromatic image to obtain the first color target image, the second color target image and the third color target image, wherein all the image pixels in the first color target image are the first color image All image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.

In some embodiments the processor 60 can be the same processor as the processor 20 arranged in the system 100 for image processing, and the processor 60 can also be arranged in the electronic device 1000, that is, the processor 60 can also be arranged in the system 100 for image processing. The processors 20 in the system 100 for image processing are not the same processor, which is not limited here.

In the description of this specification, reference to the terms “one embodiment”, “some embodiments”, “exemplary embodiments”, “example”, “specific examples” or “some examples” etc. The specific features, structures, materials or features described in the manner or example are included in at least one embodiment or example of the present disclosure.

In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or blocks of the process, and the scope of preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

Although the implementation of the present disclosure has been shown and described above, it can be understood that the above-mentioned implementation is exemplary and should not be construed as limiting the disclosure, and those skilled in the art can make changes, modifications, substitutions and variations to the above-mentioned embodiments.

Claims

1. Method for image processing, for an image sensor, the image sensor comprises a pixel array, the pixel array comprises a plurality of panchromatic photosensitive pixels and a plurality of colorful photosensitive pixels, the plurality of colorful photosensitive pixels comprise a first color photosensitive pixel, a second color photosensitive pixel and a third color photosensitive pixel with different spectral responses, the plurality of colorful photosensitive pixels have a narrower spectral response than the plurality of panchromatic photosensitive pixel, and both the first color photosensitive pixel and the third color photosensitive pixels have a narrower spectral response than the second color photosensitive pixels, the pixel array comprises a plurality of minimal repeating units, each of the minimal repeating unit comprises a plurality of subunits, each of the subunit comprises at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel; the method for image processing comprising: obtaining an original image by exposing the pixel array, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels;obtaining a colorful image based on all the colorful image pixels in the subunit, and obtaining a panchromatic image based on all the panchromatic image pixels in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; andobtaining a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel, all image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.
2. The method for image processing of claim 1, the method for image processing further comprising: obtaining a first color initial image, a second color initial image, and a third color initial image by separating the first color image pixels, the second color image pixels, and the third color image pixels in the colorful image;the obtaining a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image comprising:if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is not overexposed, calculating an updated pixel value of the image pixel to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; and if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is overexposed, calculating an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; until the target pixel values of the second color pixels corresponding to all image pixels in the second color initial image are obtained, obtaining the second color target image; andobtaining the first color target image by processing the second color target image and the first color initial image, and obtaining the third color target image by processing the second color target image and the third color initial image.
3. The method for image processing of claim 2, wherein the image pixel of the second color initial image comprises a second color image pixel and a null pixel, the pixel value of the null pixel being zero; the calculating an updated pixel value of the image pixel to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel comprising:selecting a first window centered on the image pixel to be updated in the second color initial image, and selecting a second window corresponding to the first window in the panchromatic image, the panchromatic image pixels at the center of the second window correspond to the image pixel to be updated and identified as a mapped panchromatic image pixel;obtaining a first matrix based on pixel information of all image pixels in the first window;obtaining a second matrix based on the pixel values of the mapped panchromatic image pixels, the pixel values of all panchromatic image pixels in the second window, the first matrix and a preset weight function; andobtaining the updated target pixel value of the second color pixel of the image pixel to be updated based on the pixel values of the mapped panchromatic image pixels, the pixel values of all image pixels in the first window, the pixel values of all panchromatic image pixels in the second window, and the second matrix.
4. The method for image processing of claim 3, wherein obtaining a second matrix based on the pixel values of the mapped panchromatic image pixels, the pixel values of all panchromatic image pixels in the second window, the first matrix and a preset weight function, comprising: mapping the matrix arrangement of image pixels in the second window to the array arrangement of the second matrix;obtaining the deviation value corresponding to the position of the panchromatic image pixel in the second matrix based on the pixel value of each panchromatic image pixel in the second window and the mapped pixel value of the panchromatic image pixel; andobtaining the value of the corresponding position in the second matrix based on the deviation value, a preset function and the value of the same position in the first matrix.
5. The method for image processing of claim 3, wherein obtaining the updated target pixel value of the second color pixel of the image pixel to be updated based on the pixel values of the mapped panchromatic image pixels, the pixel values of all image pixels in the first window, the pixel values of all panchromatic image pixels in the second window, and the second matrix, comprising: calculating the first weighted value based on the first window matrix formed by the pixel values of all image pixels in the first window and the second matrix, and calculating a second weighted value based on a second window matrix formed by pixel values of all image pixels in the second window and the second matrix; andobtaining the updated target pixel value of the second color pixel of the image pixel to be updated based on the mapped pixel value of the panchromatic image pixel, the first weighted value, and the second weighted value.
6. The method for image processing of claim 2, wherein the image pixel of the first color initial image comprises a first color image pixel and a null pixel, the image pixel of the third color initial image comprises a third color image pixel and a null pixel; calculating an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel, comprising: if the image pixel to be updated is the second color image pixel, identifying the value of the original pixel of the second color image pixel as the value of the target pixel of the second color pixel after updating the pixel value of the image pixel to be updated;if the position corresponding to the image pixel to be updated in the first color initial image is the first color image pixel, performing interpolation processing on the second color initial image based on the first color initial image, and obtaining the updated target pixel value of the second color pixel of the image pixel to be updated; andif the position corresponding to the image pixel to be updated in the initial image of the third color is the third color image pixel, performing interpolation processing on the second color initial image based on the third color initial image, and obtaining the updated target pixel value of the second color pixel of the image pixel to be updated.
7. The method for image processing of claim 2, wherein obtaining the first color target image by processing the second color target image and the first color initial image, and obtaining the third color target image by processing the second color target image and the third color initial image, comprising: obtaining the first color target image by performing a bilateral filtering process on the first color initial image based on the second color target image, and obtaining the third color target image by performing a bilateral filtering process on the third color initial image based on the second color target image.
8. The method for image processing of claim 1, the method for image processing further comprising: obtaining a processed colorful image by performing colorful image processing on the colorful image, obtaining a processed panchromatic image by performing panchromatic image processing on the panchromatic image;obtaining a fully arranged first color target image, a fully arranged second color target image and a fully arranged third color target image by processing of the color image based on the panchromatic image comprising: andobtain a first color target image, a second color target image and a third color target image by processing the processed colorful image based on the processed panchromatic image.
9. A system for image processing, comprising: an image sensor, the image sensor comprises a pixel array, the pixel array comprises a plurality of panchromatic photosensitive pixels and a plurality of colorful photosensitive pixels, the plurality of colorful photosensitive pixels comprise a first color photosensitive pixel, a second color photosensitive pixel and a third color photosensitive pixel with different spectral responses, the plurality of colorful photosensitive pixels have a narrower spectral response than the plurality of panchromatic photosensitive pixel, and both the first color photosensitive pixel and the third color photosensitive pixels have a narrower spectral response than the second color photosensitive pixels, the pixel array comprises a plurality of minimal repeating units, each of the minimal repeating unit comprises a plurality of subunits, each of the subunit comprises at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel; anda processor, the processor is configured toobtain an original image by exposing the pixel array, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels;obtain a colorful image based on all the colorful image pixels in the subunit, and obtain a panchromatic image based on all the panchromatic image pixels in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; andobtain a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel, all image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.
10. The system for image processing of claim 9, the system is further configured to: obtain a first color initial image, a second color initial image, and a third color initial image by separating the first color image pixels, the second color image pixels, and the third color image pixels in the colorful image;if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is not overexposed, calculate an updated pixel value of the image pixel to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; and if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is overexposed, calculate an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identify the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; until the target pixel values of the second color pixels corresponding to all image pixels in the second color initial image are obtained, obtain the second color target image; andobtain the first color target image by processing the second color target image and the first color initial image, and obtain the third color target image by processing the second color target image and the third color initial image.
11. The system for image processing of claim 10, wherein the image pixel of the second color initial image comprises a second color image pixel and a null pixel, the pixel value of the null pixel being zero; the processor is further configured to: select a first window centered on the image pixel to be updated in the second color initial image, and select a second window corresponding to the first window in the panchromatic image, the panchromatic image pixels at the center of the second window correspond to the image pixel to be updated and identified as a mapped panchromatic image pixel;obtain a first matrix based on pixel information of all image pixels in the first window;obtain a second matrix based on the pixel values of the mapped panchromatic image pixels, the pixel values of all panchromatic image pixels in the second window, the first matrix and a preset weight function; andobtain the updated target pixel value of the second color pixel of the image pixel to be updated based on the pixel values of the mapped panchromatic image pixels, the pixel values of all image pixels in the first window, the pixel values of all panchromatic image pixels in the second window, and the second matrix.
12. The system for image processing of claim 11, the processor is further configured to: map the matrix arrangement of image pixels in the second window to the array arrangement of the second matrix;obtain the deviation value corresponding to the position of the panchromatic image pixel in the second matrix based on the pixel value of each panchromatic image pixel in the second window and the mapped pixel value of the panchromatic image pixel; andobtain the value of the corresponding position in the second matrix based on the deviation value, a preset function and the value of the same position in the first matrix.
13. The system for image processing of claim 11, the processor is further configured to: calculate the first weighted value based on the first window matrix formed by the pixel values of all image pixels in the first window and the second matrix, and calculate a second weighted value based on a second window matrix formed by pixel values of all image pixels in the second window and the second matrix; andobtain the updated target pixel value of the second color pixel of the image pixel to be updated based on the mapped pixel value of the panchromatic image pixel, the first weighted value, and the second weighted value.
14. The system for image processing of claim 10, wherein the image pixel of the first color initial image comprises a first color image pixel and a null pixel, the image pixel of the third color initial image comprises a third color image pixel and a null pixel; the processor is further configured to: if the image pixel to be updated is the second color image pixel, identifying the value of the original pixel of the second color image pixel as the value of the target pixel of the second color pixel after updating the pixel value of the image pixel to be updated;if the position corresponding to the image pixel to be updated in the first color initial image is the first color image pixel, performing interpolation processing on the second color initial image based on the first color initial image, and obtaining the updated target pixel value of the second color pixel of the image pixel to be updated; andif the position corresponding to the image pixel to be updated in the initial image of the third color is the third color image pixel, performing interpolation processing on the second color initial image based on the third color initial image, and obtaining the updated target pixel value of the second color pixel of the image pixel to be updated.
15. The system for image processing of claim 10, the processor is further configured to: obtain the first color target image by performing a bilateral filtering process on the first color initial image based on the second color target image, and obtaining the third color target image by performing a bilateral filtering process on the third color initial image based on the second color target image.
16. The system for image processing of claim 9, the processor is further configured to: obtain a processed colorful image by performing colorful image processing on the colorful image, obtain a processed panchromatic image by performing panchromatic image processing on the panchromatic image; andobtain a first color target image, a second color target image and a third color target image by processing the processed colorful image based on the processed panchromatic image.
17. An electronic device, comprising: a lens;a housing; anda system for image processing, the lens and the system for image processing are couple to the housing, the lens and the image sensor of the system for image processing are configured to form an image by cooperation; the system for image processing, comprising:an image sensor, the image sensor comprises a pixel array, the pixel array comprises a plurality of panchromatic photosensitive pixels and a plurality of colorful photosensitive pixels, the plurality of colorful photosensitive pixels comprise a first color photosensitive pixel, a second color photosensitive pixel and a third color photosensitive pixel with different spectral responses, the plurality of colorful photosensitive pixels have a narrower spectral response than the plurality of panchromatic photosensitive pixel, and both the first color photosensitive pixel and the third color photosensitive pixels have a narrower spectral response than the second color photosensitive pixels, the pixel array comprises a plurality of minimal repeating units, each of the minimal repeating unit comprises a plurality of subunits, each of the subunit comprises at least one single-color photosensitive pixel and at least one panchromatic photosensitive pixel; anda processor, the processor is configured toobtain an original image by exposing the pixel array, the image pixels in the original image comprising colorful image pixels and panchromatic image pixels;obtain a colorful image based on all the colorful image pixels in the subunit, and obtain a panchromatic image based on all the panchromatic image pixels in the subunit, the image pixels in the colorful image being in a Bayer array arrangement; andobtain a first color target image, a second color target image and a third color target image by processing the colorful image based on the panchromatic image, wherein, all image pixels in the first color target image are the first image pixel, all image pixels in the second color target image are the second color image pixels, and all image pixels in the third color target image are the third color image pixels.
18. The electronic device of claim 17, the system is further configured to: obtain a first color initial image, a second color initial image, and a third color initial image by separating the first color image pixels, the second color image pixels, and the third color image pixels in the colorful image;if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is not overexposed, calculate an updated pixel value of the image pixel to be updated based on the panchromatic image and the second color initial image and identifying the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; and if the panchromatic image pixel in the panchromatic image corresponding to the image pixel to be updated in the second color initial image is overexposed, calculate an updated pixel value of the image pixel to be updated based on the first color initial image and the second color initial image, or based on the third color initial image and the second color initial image, and identify the updated pixel value of the image pixel to be updated as a target pixel value of the second color pixel; until the target pixel values of the second color pixels corresponding to all image pixels in the second color initial image are obtained, obtain the second color target image; andobtain the first color target image by processing the second color target image and the first color initial image, and obtain the third color target image by processing the second color target image and the third color initial image.
19. The electronic device of claim 18, wherein the image pixel of the second color initial image comprises a second color image pixel and a null pixel, the pixel value of the null pixel being zero; the processor is further configured to: select a first window centered on the image pixel to be updated in the second color initial image, and select a second window corresponding to the first window in the panchromatic image, the panchromatic image pixels at the center of the second window correspond to the image pixel to be updated and identified as a mapped panchromatic image pixel;obtain a first matrix based on pixel information of all image pixels in the first window;obtain a second matrix based on the pixel values of the mapped panchromatic image pixels, the pixel values of all panchromatic image pixels in the second window, the first matrix and a preset weight function; andobtain the updated target pixel value of the second color pixel of the image pixel to be updated based on the pixel values of the mapped panchromatic image pixels, the pixel values of all image pixels in the first window, the pixel values of all panchromatic image pixels in the second window, and the second matrix.
20. The electronic device of claim 19, the processor is further configured to: map the matrix arrangement of image pixels in the second window to the array arrangement of the second matrix;obtain the deviation value corresponding to the position of the panchromatic image pixel in the second matrix based on the pixel value of each panchromatic image pixel in the second window and the mapped pixel value of the panchromatic image pixel; andobtain the value of the corresponding position in the second matrix based on the deviation value, a preset function and the value of the same position in the first matrix.

Priority Claims (1)

Number	Date	Country	Kind
202011582503.6	Dec 2020	CN	national

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent Application No. PCT/CN2021/075315, filed Feb. 4, 2021, which claims the priority of Chinese Patent Application No. 202011582503.6, filed Dec. 28, 2020, both of which are herein incorporated by reference in their entireties.

Continuations (1)

	Number	Date	Country
Parent	PCT/CN21/75315	Feb 2021	US
Child	18341509		US

METHOD AND SYSTEM FOR IMAGE PROCESSING, ELECTRONIC DEVICE AND STORAGE MEDIUM

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (1)

CROSS-REFERENCE TO RELATED APPLICATIONS

Continuations (1)