The present invention relates to a method for automatic determination of values of intrinsic parameters and extrinsic parameters of a camera placed at the edge of a roadway. It also relates to a method for determining at least a physical quantity related to the positioning of said camera with respect to said roadway. It also relates to systems provided for implementing said methods. Finally, it relates to computer programs for implementing said methods.
A simplified model very widely used in the present technical field, of such a camera such as the camera 10 considers it to be a pinhole allowing a so-called perspective projection of the points Pi of the vehicle 30 on the image plane 40. Thus the equation that links the coordinates (x, y, z) of a point Pi on the vehicle 30 and the coordinates (u, v) of the corresponding point pi on the 2D image 40 can be written, in so-called homogeneous coordinates:
25
where λ is an arbitrary scalar.
The matrix [M] is a 3×4 perspective projection matrix that can be decomposed into a 3×4 positioning matrix [R T] and a 3×3 calibration matrix [K]. The calibration matrix [K] is defined by the focal distances αu and αv of the camera in terms of dimension of pixels along the axes u and v of the image 40 as well as by the coordinates u0 and v0 of the origin of the 2D image 40:
The positioning matrix [R T] is composed of a 3×3 rotation matrix and a 3-dimensional translation vector T that define, through their respective components, the positioning (distance, orientation) of the reference frame of the scene with respect to the camera 10.
For more information on the model that has just been described, reference can be made to the book entitled “Multiple View Geometry in Computer Vision” by R. Hartley and A. Zisserman, published by Cambridge University Press, and in particular to chapter 6 of this book.
In general terms, the coefficients of the calibration matrix [K] are intrinsic parameters of the camera concerned whereas those of the positioning matrix [R T] are extrinsic parameters.
Thus, in the patent application US 2010/0283856, a vehicle is used to calibrate a camera, the calibration in question being the determination of the projection matrix [M]. The vehicle in question has markers, the relative positions of which are known. When the vehicle passes in front of the camera, a 2D image is taken at a first point and another 2D image at a second point. The images of the markers in each of the 2D images are used to calculate the projection matrix [M].
The aim of the present invention is to propose a method for automatic determination of the values of intrinsic parameters and extrinsic parameters of a camera placed at the edge of a roadway. “Automatic determination” means the fact that the system is capable of determining the values of all or some of the parameters of the projection matrix [M] without implementing any particular measurement procedure and/or use of a vehicle carrying markers, such as the one that is used by the system of the patent US 2010/0283856, solely by implementing this automatic determination method.
To this end, the present invention relates to a method for automatic determination of the values of intrinsic parameters and extrinsic parameters of a camera placed at the edge of a roadway, which is characterised in that it comprises:
The present invention also relates to a method for determining at least one physical quantity related to the positioning of a camera placed at the edge of a roadway. This method is characterised in that it comprises:
The present invention also relates to a system for automatic determination of values of intrinsic parameters and extrinsic parameters of a camera placed at the edge of a roadway, which is characterised in that it comprises:
Finally, it relates to computer programs for implementing the methods that have just been described.
The features of the invention mentioned above, as well as others, will emerge more clearly from a reading of the following description of example embodiments, said description being given in relation to the accompanying drawings, among which:
a is a diagram illustrating the method for automatically determining absolute values of intrinsic parameters and extrinsic parameters of a camera according to a first embodiment of the invention,
b is a diagram illustrating a method for automatically determining absolute values of intrinsic parameters and extrinsic parameters of a camera according to a second embodiment of the invention,
The method for the automatic determination of the intrinsic and extrinsic parameters of a camera 10 (see
In a first embodiment of the invention depicted in
A second step E20 is a step of determination, by using at least one 2D image 100 of the vehicle detected at step E10 taken by the camera 10, and by using at least one predetermined 3D vehicle model 200 from a set of predetermined 3D vehicle models of different categories (for example of different models of vehicle of different makes), of the intrinsic and extrinsic parameters of the camera 10 with respect to the reference frame of the predetermined 3D vehicle model or models 200 so that a projection by the camera 10 of said or one of said predetermined 3D vehicle models 200 corresponds to said or one of the 2D images 100 actually taken by said camera 10.
According to the terminology of the present description, a predetermined 3D vehicle model is a set of points Qk of coordinates (x, y, z) in a particular reference, referred to as the reference frame. For example, the X-axis of this reference frame is a transverse axis of the vehicle, the Y-axis is the vertical axis of the vehicle and the depth axis Z is the longitudinal axis of the vehicle. As for the origin 0 of this reference frame, it is for example the projection along the Y-axis of the barycentre of said vehicle on a plane parallel to the plane (X, Z) and tangent to the bottom part of the wheels of the vehicle normally in contact with the ground. The or each predetermined 3D vehicle model is for example stored in a database 51 of the unit 50, shown in
In order to limit the number of predetermined 3D vehicle models to be used at step E20, a second embodiment depicted in
The predetermined 3D vehicle model or models {Qk} that are considered at the determination step E20 are then the predetermined vehicle model or models that were associated, at step E12, with the vehicle characteristic or characteristics recognised at step E11.
The vehicle characteristic in question here may be related to a particular vehicle (the vehicle registered xxxx), with a particular vehicle model (the vehicle brand “Simca Plein Ciel”), or a set of vehicle models (vehicles of brand Peugeot®, all models taken together).
The vehicle characteristic or characteristics that can be used, are for example, SIFT (Scale invariant feature transform) characteristics presented in the article by David G. Lowe entitled “Distinctive Image Features From Scale-Invariant Keypoints” published in International Journal of Computer Vision 60.2 (2004) p 91-110, SURF (Speed Up Robust Features) characteristics presented in the document by Herbert Bay, Tinne Tuytelaars and Luc Van Gool entitled <<SURF: Speeded Up Robust Features>> and published in 9th European Conference on Computer Vision, Graz, Austria, 7-13 May 2006, shape descriptors, etc. These characteristics may also be linked to the appearance of the vehicle (so-called Eigenface or Eigencar vectors).
Thus step E11 of the method of the invention can implement a method that is generally referred to as a “Make and Model Recognition Method”. For information on the implementation of this method, it is possible to refer to the thesis by Louka Dlagnekov already mentioned above.
The characteristic in question may also be a characteristic that unequivocally identifies a particular vehicle, for example a registration number on the number plate of this vehicle. Step E11 consists of recognising this registration number. The thesis by Louka Dlagnekov already mentioned also describes number plate recognition methods.
Two embodiments are envisaged for implementing step E20 of the automatic determination of the invention described above in relation to
In a first substep E21, a 3D model of said vehicle 30 is established from at least two 2D images 100 in a sequence of 2D images taken by the camera 10 at different instants t0 to to while the vehicle 30 detected at step E10 passes in front of the camera 10,. The 3D model in question is a model that corresponds to the vehicle 30 that is actually situated in front of the camera 10, unlike the predetermined 3D vehicle model. Such a 3D model of the vehicle 30 is a set of points Pi of coordinates taken in a reference frame related to the camera which, projected by the camera 10 at an arbitrary time, for example at time t0, form a set of points pi0 in a 2D image, denoted I0, formed by the camera 10. At a time tj, the vehicle has moved with respect to time t0 but for the camera 10 it has undergone a matrix rotation [Rj] and a vector translation Tj. Thus a point Pi on the vehicle detected is, at a time tj, projected by the camera 10 at a projection point {tilde over (p)}ij of the image Ij, such that:
{tilde over (p)}ij=K[Rj Tj]Pi
where K is a calibration matrix and [Rj Tj] is a positioning matrix.
By convention, the position of the vehicle is considered with respect to the camera at the time t0 of taking the first image in the sequence of 2D images as being the reference position so that the positioning matrix at this time t0 is then the matrix [I 0].
Next a so-called bundle adjustment method is implemented—see for example the article by Bill Triggs et al entitled “Bundle adjustment—a modern synthesis”, published in Vision Algorithms: Theory & Practice, Springer Berlin Heidelberg, 2000, pages 298 to 372, which consists of considering several points Pi of different coordinates and, from there, changing the values of the parameters of the calibration matrices [K] and positioning matrix [Rj Tj1], and, for each set of values or parameters and coordinates of points Pi, first of all determining by means of the above equation the projected points {tilde over (p)}ij and then comparing them with the points pij actually observed on an image Ij and retaining only the points Pi and the values of parameters of the positioning matrix [Rj Tj] and of the calibration matrix [K] that maximise the matching between the points {tilde over (p)}ij and the points pij, that is to say those that minimise the distances between these points. The following can therefore be written:
This equation can be solved by means of a Levenberg-Marquardt non-linear least squares optimisation algorithm.
Advantageously, the bundle adjustment method is used after a phase of initialisation of the intrinsic and extrinsic parameters and of the coordinates of the points Pi of the 3D model of the vehicle detected, in order to prevent its converging towards a sub-optimum solution while limiting the consumption of computing resources.
The intrinsic parameters of the camera may for example be initialised by means of the information contained in its technical file or obtained empirically, such as its ratio between focal length and pixel size for each of the axes of its sensor. Likewise, the main point may be considered to be at the centre of the 2D image. The values contained in this information, without being precise, are suitable approximations.
For initialising the extrinsic parameters, it is possible to proceed as follows. First of all, from a certain number of matches established between points pij of the image Ij and points pi0 of the first image I0, a so-called essential matrix E is determined that satisfies the following equation:
(K−1pij)TE(K−1pi0)=0
For more information on this process, reference can be made to the book entitled “Multiple View Geometry in Computer Vision” by R. Hartley and A. Zisserman, published by Cambridge University Press and in particular chapter 11.7.3.
Next, from this essential matrix E, the matrices [Rj Tj] are calculated for the various times tj. For more information on this process, reference can be made to chapter 9.6.2 of the same book mentioned above.
Finally, for initialising the 3D coordinates of the points Pi, it is possible to use pairs of images Ij and Ij′ and matches of points pij and pij′ in these pairs of images. The intrinsic and extrinsic parameters of the camera considered here are the parameters estimated above for initialisation purposes. For more information on this process, reference can be made to chapter 10 of the book mentioned above.
At the end of this first substep E21, a 3D model of the vehicle detected is available, defined to within a scale factor and non-aligned, that is to say of a set of points Pi of this vehicle when it is situated in the reference position mentioned above (position at time t0).
In a second substep E22, the 3D model of the vehicle detected is aligned with at least one predetermined 3D vehicle model {Qk}. In the second embodiment envisaged above in relation to
For this alignment, the parameters are sought of a geometric matrix transformation [TG] which, applied to the set or each set of points Qk of the or each predetermined 3D vehicle model, makes it possible to find the set of points Pi forming the 3D model of the detected vehicle.
The matrix [TG] can be decomposed into a scale change matrix [SM] and an alignment matrix [RM TM] where RM is a rotation matrix and TM is a translation vector. The scale change matrix [SM] is a 4×4 matrix that can be written under the form:
where sM is a scale ratio.
If a second camera that is calibrated with respect to the first camera 10 is available (it should be noted that, in this case, because cameras calibrated with each other are considered, only the extrinsic parameters of the camera 10 with respect to the road are sought), it is possible to establish, from a single pair of images, by standard stereoscopy method, a model of the detected vehicles such that sM is equal to 1.
On the other hand, if such a second camera calibrated with respect to the first camera 10 is not available, it is possible to proceed as follows. For a certain number of values of the scale ratio sM, the alignment matrix [RM TM] is determined. To do this, it is possible to use the ICP (iterative closest point) algorithm that is described by Paul J. Besl and Neil D. McKay in an article entitled “Method for registration of 3-D shapes” that appeared in 1992 in “Robotics-DL Tentative”, International Society for Optics and Photonics.
For each value of the scale ration sM, an alignment score s with a good match is established, for example equal to the number of points Pi that are situated at no more than a distance d from points Pk such that:
∥PiPk∥<d with
Pk=SM[RM TM]Qk
Next the scale ratio value sM and the corresponding values of the parameters of the alignment matrix [RM TM] that have obtained the best alignment score s are selected. This is the best alignment score s.
If several predetermined 3D vehicle models are available, as before for a single predetermined 3D vehicle model, the best alignment score s is determined this time for each predetermined 3D vehicle model and then the predetermined 3D vehicle model that obtained the best score on best alignment is adopted. The predetermined 3D vehicle model adopted corresponds to a vehicle model that can in this way be recognised.
In a third substep E23, the extrinsic parameters of the camera with respect to the reference frame of the predetermined 3D vehicle model of the vehicle recognised are determined. To do this the following procedure is followed.
For each point pk0 of the 2D image I0 delivered by the camera 10 at time t0, there is a corresponding point Qk in the predetermined 3D vehicle model, so that the following can be written:
pk0=KSM[RM TM]Qk=K[RM TM]Qk
Thus the matrix of extrinsic parameters of the camera relative to the predetermined 3D vehicle model of the vehicle recognised is the matrix:
[R T]=[RM TM]
A description is now given in relation to
For this embodiment, each predetermined 3D vehicle model 200 in said set of predetermined 3D vehicle models for example stored in the database 51 consist not only of a proper predetermined 3D vehicle model 201, that is to say a set of points Qk, but also points pk of at least one 2D reference image 202 obtained by projection, by a reference camera, real or virtual, of the points Qk of the predetermined 3D vehicle model 201. Thus, for each predetermined 3D vehicle model 200, the points pk of the or each reference image 202 match points Qk of said predetermined 3D vehicle model proper 201 (see arrow A).
There is also available a 2D image 100 actually taken by the camera 10 of the vehicle 30 detected at step E10 of the method of the invention (see
In a first substep E210, matches between points pi of the 2D image 100 of the vehicle 30 detected and points pk of the image or of a reference 2D image 202 (arrow B) are first of all established and then, in a second substep E220, matches between points pi of the 2D image 100 of the vehicle 30 detected and points Qk of the predetermined 3D vehicle model proper 201 considered (arrow C). As before, in the second embodiment envisaged above in
It is considered that each point pi of the 2D image 100 is the result of a transformation of a point Qk of the proper predetermined 3D vehicle model 201 of the vehicle 30 detected. This transformation can be assimilated to a projection made by the camera 10, an operation hereinafter referred to as “pseudo-projection”, and it is thus possible to write:
where [A] is a 3×4 matrix then said to be pseudo-projection.
If a sufficient number of matches are available (generally at least 6 matches), this equation represents an overdetermined linear system, that is to say where it is possible to determine the coefficients of the pseudo-projection matrix [A]. This calculation of the matrix [A], carried out at step E230, is for example described in chapter 7.1 of the book mentioned above. At the following step E240, the intrinsic and extrinsic parameters of said camera are deduced from the parameters thus determined of said pseudo-projection matrix [A].
The pseudo-projection matrix [A] can be written in the following factorised form:
[A]=K [R T]=[K[R]KT]
where [R] is the matrix of the rotation of the camera 10 with respect to the predetermined 3D vehicle model of the recognised vehicle and T the translation vector of the camera with respect to the same predetermined 3D vehicle model.
The 3×3 submatrix to the left of the pseudo-projection matrix [A] is denoted [B].
This gives:
[B]=K[R].
The following can be written:
[B][B]T=K[R](K[R])T=K[R][R]TKT=KKT
If it is assumed that the calibration matrix K is written under the form of the above equation (2), it is possible to write, by developing KKT:
The product [B][13]T can be written under the form of these coefficients:
[B][B]T=[bij] with i, j=1 to 3.
From knowledge of [B][B]T=λ KKT obtained from the matrix [A], it is possible to calculate λ (the parameter λ is then equal to b33) and the coefficients of the calibration matrix K, and then the parameters of the matrix [R T]=K−1 [A].
Whereas the first embodiment (see
Once the intrinsic and extrinsic parameters of the camera have been determined with respect to the reference frame of the predetermined 3D vehicle models (the reference frame is identical for all the 3D models) stored in the database 51, and thus when a vehicle 30 having remarkable characteristics that can be recognised at step E11 passes in front of the camera 10, it is possible to determine a certain number of physical quantities.
Thus the present invention concerns a method for determining at least a physical quantity related to the positioning of a camera placed at the edge of a roadway. It comprises (see
It also comprises a step E2 of establishing, from said parameter values, the positioning matrix of the camera [R T] and then, at a step E3, of calculating the matrix [R′ T′] of the inverse transformation.
Finally, it comprises a step E4 of deducing, from said positioning matrix [R T] and the inverse transformation matrix [R′ T′], the or each of said physical quantities in the following manner:
the height h with respect to the road: h=T′z.
Two quantities remain unknown :
Number | Date | Country | Kind |
---|---|---|---|
14/56767 | Jul 2014 | FR | national |