The present disclosure generally relates to the field of image processing, and more particularly aims at a system and a method for generating the descriptors of a scene captured by an image sensor.
In the field of image processing, there exist many applications where descriptors enabling to highlight information of interest in the image are generated, for example, to detect predetermined elements in the image and/or to implement classification operations.
The generation of descriptors in a sensor capable, during an acquisition phase, of successively supplying a plurality of binary image planes, for example, a sensor of the type described in patent applications EP1119311 and US20180124338 previously filed by the applicant.
An embodiment provides a scene descriptor generation system, comprising:
an image sensor comprising a plurality of pixels, the sensor being configured to, during an acquisition phase, successively supply S binary image planes, each comprising a single binary value per pixel, S being an integer greater than or equal to 2; and
a processing circuit configured to, for each binary image plane supplied by the sensor, implement the successive steps of:
According to an embodiment, the processing circuit is configured so that the N convolved images provided at step a) and said at least one meta-image generated at step b) have same dimensions, and so that each pixel value of said at least one meta-image is equal to the result of a mathematical function having as inputs the N values of pixels of same position in the N convolved images.
According to an embodiment, the processing circuit is configured so that the mathematical function comprises at least one statistical function from the group comprising the maximum, the average, the median, the minimum, and the standard deviation.
According to an embodiment, the processing circuit is configured so that the N binary descriptors supplied at step c) are binary images of same dimensions as the N convolved images supplied at step a) and as said at least one meta-image generated at step b).
According to an embodiment, the processing circuit is configured so that, for each of the N binary descriptors supplied at step c), each pixel value of the binary descriptor is equal to the result of a logic function with a binary output, the logic function receiving as an input only values from the group comprising the N values of pixels of same position in the N convolved images and the value of the pixel of same position in said at least one meta-image.
According to an embodiment, the processing circuit is further configured to, for each binary image plane supplied by the sensor, after step c), implement the step of:
According to an embodiment, the processing circuit is configured to, after step c), repeat steps a), b), and c) b replacing the input binary image plane With one or a combination of, all or part of the binary descriptors generated at step c).
According to an embodiment, the processing circuit is configured to, at the end of step d), repeat steps a), b), and c) by replacing the input binary image plane with one or a combination of all or part of the binary descriptors of decreased dimensions generated at step d).
According to an embodiment, the pixels of the sensor are arranged in an array of rows and columns, the sensor further comprising a control circuit configured to successively select the pixels, row by row, according to a control method of rolling shutter type.
According to an embodiment, the control circuit of the sensor further comprises a quantization circuit capable of comparing an output signal of each pixel of the selected row with a threshold and of delivering a binary output value representative of the result of the comparison.
Another embodiment provides a method of generation, by means of a processing circuit, of descriptors of a scene captured by an image sensor comprising a plurality of pixels, the sensor being configured to, during an acquisition phase, successively supply S binary image planes, each comprising a single binary value per pixel, S being an integer greater than or equal to 2, the method comprising the successive steps of:
The foregoing and other features and advantages will be discussed in detail in the following non-limiting description of specific embodiments in connection with the accompanying drawings, in which;
Like features have been designated by like references in the various figures. In particular, the structural and/or functional features that are common among the various embodiments may have the same references and may dispose identical structural, dimensional and material properties.
For the sake of clarity, only the steps and elements that are useful for an understanding of the embodiments described herein have been illustrated and described in detail. In particular, the present disclosure essentially concerns a system and a method for generating descriptors of a scene captured by an image sensor. The uses that can be made of such descriptors and the various descriptor parameterizing possibilities have not been detailed, the described embodiments being compatible with all or most known applications using image descriptors, for example, for applications of automatic detection of an element in an image, and the parameterizing of the descriptors according to the considered application being within the abilities of those skilled in the art.
Further, in the examples described hereafter, the forming of the internal circuits of the image sensors, and particularly of the pixels and of the peripheral control and readout circuits, has not been detailed. Further, the forming of a processing circuit capable of implementing the provided descriptor generation method has not been detailed, the forming of such a circuit being within the abilities of those skilled in the art based on the indications of the present disclosure. It should be noted that the processing circuit may be partially or totally integrated hi the same semiconductor chip as the image sensor, or may be integrated in a semiconductor chip external to the image sensor.
Unless indicated otherwise, when reference is made to two elements connected together, this signifies a direct connection without any intermediate elements other than conductors, and when reference is made to two elements coupled together, this signifies that these two elements can be connected or they can be coupled via one or more other elements.
Unless specified otherwise, the expressions “around”, “approximately”, “substantially” and “in the order of” signify within 10%, and preferably within 5%.
The sensor of
In the representation of
In this example, the pixel rows are successively selected according to a control method of rolling shutter type. All the pixel rows of the sensor are successively scanned S times. For each selection of a pixel row, the pixels in the row are simultaneously quantized (in parallel) by quantization circuit 103(b). At the end of each scanning of the L* rows of the sensor, a binary image plane Ps of dimension. C**L* is obtained, s being an integer in the range from 1 to S. Each binary image plane Ps comprises a single binary value per pixel. In
As an example, for each pixel, the S successive selections of each pixel row are performed during a same phase of integration of the pixel row. The quantization threshold applied to each pixel in the row by circuit 103(b) may be constant all along the integration phase. For a given pixel, the position of the transition edge between the low state and the high state in the binary output sequence is representative of the light intensity received by the pixel during the integration phase. The time interval between two successive selections of each pixel row is for example substantially constant all along the pixel row integration phase. As a variation, the time interval between two successive selections of each pixel row varies during the integration phase in a way which is inversely proportional to die rank of the first one of the two selections among the S successive selections of the pixel row.
As a variation, die S successive selections of each pixel row, instead of being performed during the phase of integration of the pixels in the row, are performed during a readout phase following the phase of integration of the pixel row. The quantization threshold applied to each pixel in the row by circuit 103(b) may then vary monotonously all along the readout phase.
More detailed embodiments of a sensor of the type described in relation with
The system of
It is here desired to generate, for each binary image plane Ps supplied by sensor 100, a plurality of descriptors enabling to highlight information of interest in the image, for example, to detect predetermined elements in the image and/or to implement classification operations.
For this purpose, the system of
The processing method of
A routing circuit, not detailed in
For the rest of the disclosure, it is considered that at each scanning of the L* sensor rows, processing circuit 200 receives a binary image plane Is of dimensions C×L corresponding to the complete binary image plane Ps (that is, C=C* and L=L*), or to a portion of binary image plane Ps, C, and L being integers greater than or equal to 2 respectively designating the number of columns and the number of rows of binary image plane Is. In practice, processing circuit 200 may be configured to, at each scanning of the L* sensor rows, receive and process in parallel a plurality of binary image planes Is of dimensions C×L.
The processing method of
In this example, the N convolved images Xn obtained at the end of step 301 are non-binary images.
The method of
The K meta-images Zk calculated at step 302 are for example of same dimensions C′×L′ as the N convolved images Xn calculated at step 301. As an example, for each meta-image Zk, each pixel value of meta-image Zk is equal to the result of a mathematical function fk having as inputs the N values of pixels of same position in the N convolved images Xn. Functions fk are preferably statistical functions, for example, the average, the median, the maximum, the minimum, the standard deviation, etc.
As an example, in the case where step 301 is implemented in accordance with the example of
Z1(c′,l′)=f1(Xn(c′,l′))=man(max(Xn\5(c′,l′))2), [Eq. 10]
that is:
Z1(c′,l′)=max(max(X1(c′,l′),X2(c′,l′),X3(c′,l′)X4(c′,l′), X6(c′,l′),X7(c′,l′),X8(c′,l′),X9(c′,l′)),2) [Eq. 11]
In the above-mentioned equations Eq. 10 and Eq. 11, c′ is an integer in the range from 1 to C′, I′ being an integer in the range from 1 to L′, Z1(c′,l′) designates the value of the pixel of position c′,I′ in meta-image Z1, Xn(c′,I′) designates the value of the pixel of position c′,l′ in image Xn, and Xn\5 designates the assembly of images Xn except for image X5 (n but 5).
The method of
XTn(c′,l′)=ln(X1(c′,l′), . . . , XN(c′,l′),Z1(c′,l′),Z1(c′,l′), . . . , ZK(c′,l′)) [Eq. 12]
Logic functions in may be functions with a signed binary output, or functions with an unsigned binary output.
As an example, in the case where step 301 is implemented in accordance with the example of
XTn\5(c′,l′)=((X5(c′,l′)−Xn(c′,l′)≤2)&&(Xn(c′,l′)≥Z1(c′,l′)) Eq. 13
and:
XT5(c′,l′)=(X5(c′,l′)≥5) [Eq. 14]
In other words, for any n but n=5, XTn(c′,l′) is in a high state if difference X5(c′,l′)−Xn(c′,l′) is smaller than or equal to 2 and if value Xn(c′,l′) is greater than or equal to Z1(c′,l′), and in a low state in the opposite case. For n=5, XT5(c′,l′) is in a high state if value X5(c′,l′) is greater than or equal to 5, and in a low state in the opposite case.
The method of
As an example, the pixel sub-groups are groups of (g=8)×(l=8) neighboring pixels. Pooling function p is for example a function of maximum-median-maximum type comprising, in each group of 8×8 pixels:
It should be noted that steps 301, 302, 303 and, possibly, 304, may be repeated by re-injecting at the input of the processing circuit binary descriptors XTn or YTn or a combination of binary descriptors XTn or YTn (instead of binary image planes Is).
The method of
For each index n in the range from 1 to N, the S binary descriptors XTn or YTn of same index n may be digitally integrated, that is, added point by point, as they are supplied by processing circuit 200 to obtain, at the end of the acquisition phase, N non-binary descriptors of dimensions C′×L′ or (C′/g)×(L′/h). Such non-binary descriptors may then be used for applications of analysis of various scenes, for examples, applications of detection of predetermined elements in the scene and/ex to implement classification operations.
As a variation, after each iteration of the method of
Examples of on-the-fly bitstream processing methods are for example described in above-mentioned patent applications EP3319311 and US20180124348. It should further be noted that it will be within the abilities of those skilled in the art to provide different variants and topologies of neural networks according to the type of inference problem to be addressed.
Various embodiments and variants have been described. Those skilled in the art will understand that certain features of these various embodiments and variants may be combined, and other variants will occur to those skilled in the art. In particular, the described embodiments are not limited to the above-described examples of parameterizing of the descriptor generation method. In particular, other convolution kernels □n as those of the example of
Finally, the practical implementation of the embodiments and variants described herein is within the capabilities of those skilled in the art based on the functional indications provided hereinabove. In particular, the forming of processing circuit 200 capable of implementing the processing method described in relation with
Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and the scope of the present invention. Accordingly, the foregoing description is by way of example only and is not intended to be limiting. The present invention is limited only as defined in the following claims and the equivalents thereto.
Number | Date | Country | Kind |
---|---|---|---|
FR1907662 | Jul 2019 | FR | national |
Number | Name | Date | Kind |
---|---|---|---|
20130163870 | Cao | Jun 2013 | A1 |
20150092982 | Kang | Apr 2015 | A1 |
20160225119 | Xu | Aug 2016 | A1 |
20190019307 | Dupoiron | Jan 2019 | A1 |
20210012107 | Verdant | Jan 2021 | A1 |
Number | Date | Country |
---|---|---|
3428851 | Jan 2019 | EP |
WO-2012136642 | Oct 2012 | WO |
Entry |
---|
Preliminary Search Report for French Application No. 1907662 dated Apr. 16, 2020, 2 pages. |
Number | Date | Country | |
---|---|---|---|
20210012107 A1 | Jan 2021 | US |