Patent Grant
12039755
The present invention relates to a method and device for estimating the distance (or ranging) between an observer and a target, using at least one image generated by a digital image generator.
Although not exclusively, the present invention is particularly applicable to the military field. In the military field, the estimation of the distance of a target or a threat (relative to the observer) is very important. It is of particular interest:
In the scope of the present invention, a target is any object, for example a building or a vehicle, or any other element, whose distance is to be measured.
Different systems, based on different and various technologies, are known to measure the distance between an observer and a target, i.e. between the position of the observer and that of the target.
In particular, methods based on the analysis of a light signal emitted by the observer and reflected by the observed target are known. Such a method exists in active systems such as a laser rangefinder, a 3D scanner, a lidar or time-of-flight cameras. The distance measurement is performed by different analyses, implemented in these systems, such as the delay measurement of the echo of the emitted signal, the measurement of the phase shift between the emitted and reflected signals, the measurement of a frequency modulation, and the measurement of the decrease of the intensity of the reflected signal.
All the active methods have disadvantages in terms of stealth. Indeed, they require the emission of an electromagnetic radiation towards the observed target. This radiation is detectable, which is not desirable in some applications. In addition, it requires specific active equipment, which may be expensive.
There are also passive methods, in particular focusing methods of a shallow depth of field imager. Such a method requires a specific equipment that is poorly adapted to other uses that could be envisaged with the same optical device, such as the observation or the surveillance of an area.
Other passive methods based on triangulation also have disadvantages. A typical device uses generally two imagers, which is problematic in terms of cost and space requirements. In addition, the performance is directly related to the distance between the two imagers, which is particularly unfavorable for the space requirements. A variant with a single moving imager requires the observer to move, which is not always possible.
These usual solutions are therefore not completely satisfactory.
The object of the present invention is to provide a particularly advantageous passive distance measurement (or ranging).
The present invention relates to a method for estimating a distance between a position of an observer and a target, using at least one image generated by a digital image generator (or imager or camera) from the position of the observer, which allows to remedy the aforementioned disadvantages.
To this end, according to the invention, said distance estimation method comprises a sequence of steps comprising:
Thus, thanks to the invention, a method is obtained which allows to estimate, automatically and without contact, the distance separating an object (target) and an observer, without the latter having to move, and this by means of an optical system (image generator) which is passive, i.e. which does not emit electromagnetic waves in the direction of the target to interact with it.
In a preferred embodiment, the detection and identification step comprises:
Advantageously, the detection sub-step consists in performing one of the following operations on the image: identifying the pixels of the target, identifying the pixels of the outline of the target, generating a bounding box encompassing the target. A learning technique for the detection can be used advantageously (but not necessarily and independently of the identification models).
In addition, advantageously, the detection and identification step implements a machine learning using classification models.
Advantageously, the method comprises at least one learning step, implemented prior to the detection and identification step, consisting in generating said classification models. Further, in a particular embodiment, said learning step implements one of the following algorithms:
Furthermore, in a preferred embodiment, the comparison sub-step consists in assigning a score to each of said projected representations, according to the quality of the match (measured by a similarity measure between the imaged representation and the projected representation), and the selection sub-step consists in selecting the projected representation with the best score. Advantageously, the quality of the match corresponds to the measure of the integration along the projected outline of the distance between the outline of the imaged representation and the outline of the projected representation.
Furthermore, advantageously, said three-dimensional models of targets are geometric representations of the targets as polygons in space. Advantageously, the method comprises at least one generation step, implemented prior to the detection and identification step, consisting in generating said three-dimensional models.
In a particular embodiment, the method further comprises at least one of the following steps:
The method can be implemented in two different ways. Specifically:
In this second embodiment, it is envisaged that the operator can independently intervene on the segmentation of the object and on the identification class of the object, using a man/machine interface (for example a software with a graphic interface implemented on a touchscreen or with a cursor). For example, the operator helps the device to draw the outlines of the object, or he informs the type of object he has extracted from the image (e.g. a tank of a given type).
The present invention also relates to a device for estimating the distance between a position of an observer and a target, using at least one image generated by a digital image generator from the position of the observer.
According to the invention, said (distance estimation) device comprises:
Further, in a particular embodiment, said device also comprises at least one of the following elements:
The figures of the attached drawing will make it clear how the invention can be carried out. In these figures, identical references designate similar elements.
The distance estimation device 1 (hereinafter “device”) shown schematically in
Target 4 means any object, e.g. a building, a mobile or non-mobile vehicle or any other element, the distance of which is to be measured (relative to the observer 2). In the example shown in
The device 1 is intended to estimate the distance with the aid of at least one image I (
More precisely, the device 1 is intended to estimate the distance D between the current position P1 of the observer 2 (from which the image I was taken) and the current position P2 of the target 4, as shown in
For the implementation of the invention, technical characteristics of the image generator 6 are known, and in particular the spatial resolution of said image generator 6, namely the angular value of the angle of view (or viewing angle) in the reality which corresponds to the length of a pixel in the image I generated by said image generator 6.
In the scope of the present invention, the observer 2 and/or the target 4 do not need to be mobile to implement the distance estimation. However, the observer 2 may be movable or stationary, and the target 4 may be movable or stationary, for the implementation of the present invention.
The device 1 comprises, as shown in
Imaged representation 22 of a target 4 in an image I means the representation (or image part) representing that target 4 in the image I. For simplification purposes, only one target 4 of imaged representation 22 is shown in image I of
The device 1 further comprises at least one predetermined database 14, containing at least said predetermined list of known targets. In a particular embodiment, the device 1 comprises a set (not shown) of a plurality of databases 14, only one of which is shown in
The known targets may be different versions or different models of the same type of sought vehicle or object, for example a mobile vehicle, in particular land-based, such as a tank 5, a military truck or the like. It can also be different types of targets (tank, transport vehicle, etc.).
In a first embodiment shown in
The device 1 further comprises at least one predetermined database 20 containing three-dimensional (3D) models of known targets, as specified below. These targets represent the targets taken into account in the database 14. In a particular embodiment, the device 1 comprises a set (not shown) of a plurality of databases 20, only one of which is shown in
In a first embodiment shown in
In one embodiment, the device 1 comprises a single database (not shown), and the data from the databases 14 and 20 are integrated into this single database.
In a preferred embodiment, the detection and identification unit 8 comprises:
The detection and identification unit 8 thus comprises a processing chain of the DRI type (for “Detection, Recognition and Identification”) which has the function of detecting and locating, as well as identifying the objects (or targets) of the database 14 which are possibly present in the image I.
In addition, the distance estimation unit 10 comprises:
Furthermore, in a particular embodiment, said device 1 also comprises the image generator 6 which is thus configured to generate at least one image I from the position P1 of the observer 2.
In a preferred embodiment, the image generator 6 is an optoelectronic imager, for example an RGB (for “Red, Green, Blue”) camera or an infrared camera.
Furthermore, in a particular embodiment, said device 1 comprises an information transmission unit (not shown) which is configured to transmit at least the estimated distance D (as well as the identity of the target for example) to at least one user system (not shown) via a link 19 (
The device 1, as described above, implements a distance estimation method P (hereinafter “method”) shown in
Said method P comprises, as shown in
Said method P also comprises, as shown in
In a preferred embodiment, the detection and identification step E1 comprises:
The detection sub-step E1A consists in performing one of the following operations on the image: identifying the pixels of the outline of the target or the pixels of the target or generating a bounding box encompassing the target.
The detection sub-step E1A aims to segment the target 4 finely, i.e. to distinguish it from the background F of the image I (
To this end, in a first embodiment shown in
Furthermore, in a second embodiment shown in
The purpose of the detection and identification step E1 is thus to detect and locate, as well as to identify the targets of the database 14, possibly present in the image I.
This detection and identification step E1 thus comprises two main functions (or sub-steps), namely:
In both sub-steps E1A and E1B, the methods considered for performing the functions are based on machine learning techniques. The machine learning uses models built beforehand during a learning phase. This learning phase is performed offline, i.e. it is implemented only once before using the device 1 in a second online (or test) phase on observation images. During this phase, a learning algorithm builds models that are then used in the test phase.
They are called classification models because they allow the device to classify the target examples extracted from the observed scene:
The method P comprises at least one learning step EOA, implemented prior to the detection and identification step E1, consisting in generating the (classification) models stored in the database 14.
The proposed learning algorithm uses to build the models, a base of example images, whose ideal response by the classification models (detection and recognition and identification classes) is known. The classes are predetermined. This is called tagged data. Such a learning technique using data labeled by an expert is referred to as supervised learning. The base of labeled images used by the learning is referred to as the training database. It can potentially comprise a large number of images.
By way of illustration, we can cite some examples of supervised learning-based algorithms that can be used in the learning step EOA (
Therefore, the inputs to the detection and identification unit 8 (implementing the detection and identification step E) are as follows:
The outputs of the detection and identification unit 8 (implementing the detection and identification step E1), which are transmitted to the distance estimation unit 10, are as follows:
The detection and identification unit 8 thus provides a segmentation of the target 4 and its label in the database 14.
The processing described in the present description for estimating the distance of a target 4 identified in an image I can of course be implemented (in a similar manner each time) for estimating the distances of each of a plurality of targets, in the case where several (known) targets are identified in the image I.
Once the target is finely located in the image I and identified, the method P comprises the distance estimation step E2 implemented by the distance estimation unit 10.
The distance estimation step E2 comprises a projection sub-step E2A, implemented by the projection element 12, consisting in performing on the image I a plurality of different projections of a three-dimensional model of said identified target 4.
This plurality of different projections of the model M (in three dimensions) on the image I allows to obtain projected representations 29 and 30 of the target 4, as represented in
Each of said projected representations 29 and 30 is representative of a particular given (associated) distance of the target 4 with respect to the image generator 6 and a particular given (associated) orientation of the target 4 in space. The orientation can be changed by moving the model M around and/or along one of the X, Y and Z directions of a frame of reference R represented illustratively in
The distance estimation step E2 also comprises a comparison sub-step E2B, implemented by the comparison element 13, consisting in comparing the imaged representation 22 of the target 4 with each of said projected representations 29 and 30, as shown for the projected representation 29 in
Furthermore, the distance estimation step E2 comprises a selection sub-step E2C, implemented by the selection element 15, consisting in selecting, among the set of projected representations 29 and 30 compared in the comparison step E2B to the imaged representation 22, the projected representation most similar to said imaged representation 22, namely the projected representation 30 in our example.
To this end, in a preferred embodiment, the comparison sub-step E2B assigns a score to each of said projected representations 29 and 30, based on the quality of the match, and the selection sub-step E2C selects the projected representation with the best score, where the better the match (or the quality of the match) the higher the score.
The quality of the match (for a given or considered projected representation) is the result of the measurement of the integration (along the outline of the considered projected representation) of the distance between the outline of the imaged representation and the outline of said considered projected representation. In order to have a good match and thus a high score, it is convenient to have a distance (integrated along the outline) between the two outlines (of the imaged representation and of the projected representation) as small as possible, i.e. the projected representation should be as similar as possible to the imaged representation.
The distance associated with the selected projected representation 30 represents said estimated distance D between the position of the image generator 6 (and thus the observer 2) and that of the target 4.
The distance estimation step E2 thus consists in comparing the target 4 observed in the pixels of the image I (imaged representation 22) with a geometric modeling of this target 4 (comprising its shape in three dimensions and the measurements of its dimensions in meters). By comparing the pixel dimensions of the imaged representation 22 of the target 4 with the dimensions in meter of its geometric model, an estimation of the distance between the target 4 and the observer 2 can be produced, knowing the angular resolution of the image generator 6.
The distance estimation step E2 is based on the use of 3D models of known targets. Therefore, a 3D model is available for each of the targets in the knowledge base (database 14), which we have been able to segment and identify, and which we wish to range. We use a transformation of the 3D model whose parameters provide an estimation of the distance sought. A transformation of this model is used, depending on the observation image of the target that is being processed. This transformation is the projection of the 3D model into the image. This is the projection of the target in 3D, the result of which is the image of this target seen by the image generator 6 if it had been in the observed scene at a certain (associated) distance and with a certain (associated) orientation. The parameters of this projection are therefore the geometric characteristics of the image generator 6 used, the orientation (or pose) in space of the model, and the distance at which the target thus observed would be located. The result of this projection is a two-dimensional profile (or outline or silhouette) in the image I, corresponding to said projected representations 29 and 30.
The model is therefore projected according to different orientation and distance parameters in order to obtain a profile (or outline) of the projected model that is as close as possible to the profile (or outline) of the target actually observed in the image I (imaged representation 22). The best projection is therefore sought, which also requires the construction of a score for this projection to assess the extent to which the projected profile corresponds to the observed target. The best projection score obtained after exploring several projection parameters provides the likely distance between the target and the observer. In other words, the best set of parameters (distance, orientation) obtained provides the distance estimation.
Preferably, the 3D models are geometric representations of the known targets as polygons in space. They are built from the knowledge of the geometry of the targets in the database 14.
The method P also comprises at least one generation step EOB, implemented prior to the detection and identification step E1, consisting in generating said 3D models stored in the database 20.
The method P, as described above, can be implemented in two different ways.
In a first embodiment, it is implemented fully automatically.
In a second embodiment, the detection and identification step E1 is implemented semi-automatically (with an intervention of an operator). In this case, the operator imposes or corrects the segmentation and/or the identification of the target. In addition, the distance estimation step E2 is implemented automatically.
The method P also comprises at least one image generation step EOC, implemented prior to the detection and identification step E1, consisting in generating at least one image I using the image generator 6 from the position P1 of the observer 2.
In a particular embodiment, the device 1 comprises additional distance calculation elements (not shown) based on several techniques used in parallel. The generated result (estimated distance) is obtained, in this embodiment, from a combination (average, etc.) of the results of these different techniques. The complementarity of the different techniques can thus be used to improve the overall robustness and accuracy of the estimation of the distance.
The method P thus allows to estimate the distance D between a target 4 observed in a digital image I and the image generator 6 (which generated this digital image I) which is geometrically characterized (calibrated). More particularly, the processing chain implemented by the method P allows a distance measurement between the known observed target and the observer, from a single image (a single camera, at a given time) and without electromagnetic emissions intended to illuminate the target (passive system).
The method P allows to estimate, automatically and without contact, the distance separating a target and its observer without the latter having to move, by means of a passive optical system (image generator 6).
In summary, the method P is limited to the measurement of the distance between the observer 2 and a set of previously known and geometrically characterized targets 4. For this purpose, a database of targets is available, the dimensions of which are known in particular, and an attempt is made to measure the distance D between the observer 2 and these targets 4 when they are observed in a digital image I resulting from the image generator 6 used. The image generator 6 is calibrated. In particular, its angular resolution is known. When the observation image I is generated, it is subjected to a processing chain (method P), having two main steps.
The method P (of distance estimation), as described above, has the following advantages in particular:
| Number | Date | Country | Kind |
|---|---|---|---|
| 1906854 | Jun 2019 | FR | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/FR2020/050875 | 5/26/2020 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2020/260782 | 12/30/2020 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 9300946 | Do | Mar 2016 | B2 |
| 9361660 | Tanaka | Jun 2016 | B2 |
| 11170561 | Baker | Nov 2021 | B1 |
| 20100092079 | Aller | Apr 2010 | A1 |
| 20120262455 | Watanabe | Oct 2012 | A1 |
| 20190147221 | Grabner et al. | May 2019 | A1 |
| 20230057655 | Du | Feb 2023 | A1 |
| Entry |
|---|
| International Search Report issued in corresponding International Patent Application No. PCT/FR2020/050875 dated Sep. 15, 2020. |
| Number | Date | Country | |
|---|---|---|---|
| 20220351412 A1 | Nov 2022 | US |
