Patent Application
20180164122
This application is a U.S. non-provisional application claiming the benefit of French Application No. 16 62264, filed on Dec. 9, 2016, which is incorporated herein by reference in its entirety.
The present invention relates to an electronic device for controlling a drone. The electronic device comprises a first acquisition module configured for acquiring a succession of images of a terrain overflown by the drone, and taken by an image sensor equipping the drone, and a second acquisition module configured for acquiring a measured ground speed via a measuring device equipping the drone.
The electronic device further comprises a calculation module configured for calculating an altitude of the drone with respect to the terrain, based on an optical flow algorithm applied to the acquired images and the acquired measured ground speed.
The invention also relates to a drone comprising an image sensor and such an electronic control device.
The invention also relates to a method for controlling the drone.
The invention also relates to a non-transitory computer-readable medium including a computer program comprising software instructions which, when executed by a computer, implement such a control method.
The invention relates to the field of drones, i.e. remote-controlled motorized flying devices. The invention applies, in particular, to fixed-wing drones, while also applying to other types of drones, for example rotary wing drones, such as quadricopters.
It is known, from the article “Determining Altitude AGL Using Optical Flow” by Jonathan Price, that a drone comprises a control device of the aforementioned type. The drone is a fixed-wing drone, which is controllable by using a portable electronic device, such as a smartphone or an electronic tablet.
The electronic device is configured for acquiring a succession of images of a terrain overflown by the drone, and taken by an image sensor equipping the drone, and also a measured ground speed, provided by a measuring device equipping the drone, such as a satellite positioning system, also known as a Global Positioning System (GPS).
The control device is configured for calculating an altitude of the drone with respect to the terrain, based on an optical flow algorithm applied to the acquired images and the measured ground speed.
The altitude thus calculated verifies the following equation:
Altitude=Vsol_mes/(Optical Flow−Pitch Rate)
However, the altitude thus calculated is not always very reliable, and the operation of the drone reflects this.
The object of the invention is therefore to propose an electronic control device that calculates more reliably the altitude of the drone with respect to the terrain overflown, and thus allows reducing possible jerkiness during the operation of the drone, in particular in the landing phase.
For this purpose, the subject-matter of the invention is an electronic control device of the aforementioned type, wherein the second acquisition module is configured for further acquiring an altitude of the drone measured with respect to a reference level, and the device further comprises a recalibration module configured for correlating the altitude calculated with respect to the terrain with the altitude measured with respect to the reference level.
The electronic control device according to the invention thus allows recalibrating the altitude of the drone calculated with respect to the terrain, by making a correlation between the altitude calculated with respect to the terrain and the altitude measured with respect to the reference level. The reference level is, for example, sea level, and the altitude measured with respect to sea level is, for example, obtained via a pressure sensor.
The measured ground speed is, for example, provided by a satellite positioning device, also called GNSS (Global Navigation Satellite System), such as a GPS (Global Positioning System) receiver, and/or by an inertial sensor.
The altitude with respect to the terrain, thus calculated and recalibrated, is then more reliable, and is particularly useful, especially in the landing phase, to better predict the approach to the ground, and to anticipate the moment when the control device has to control the drone's pitch, i.e. an increase in its pitch, in order to land.
According to other advantageous aspects of the invention, the electronic control device comprises one or more of the following features, taken separately or in any technically possible combination:
The subject-matter of the invention is also a drone comprising an image sensor configured for taking a succession of images of a terrain overflown by the drone, and an electronic control device, wherein the electronic control device is as defined above.
The subject-matter of the invention is also a method for controlling a drone comprising an image sensor, wherein the method is implemented by an electronic device and comprises:
The invention also relates to a non-transitory computer-readable medium including a computer program comprising software instructions which, when executed by a computer, implement a method as defined above.
These features and advantages of the invention will appear more clearly upon reading the description which follows, given solely by way of a non-limiting example, and with reference to the appended drawings, wherein:
In the following of the description, the expression “substantially constant” means a value plus or minus 10%, i.e. with a variation of at most 10%, more preferably as a value plus or minus 5%, i.e. with a variation of at most 5%.
In
The drone 10 comprises a second image sensor 18 configured for taking images of a scene towards which the drone 10 is moving, the second image sensor 18 being for example a forward-facing camera.
The drone 10 also comprises an altimeter 20, such as a radioaltimeter or an ultrasonic rangefinder, emitting a beam 22 towards the ground making it possible to measure the altitude of the drone 10 with respect to the terrain 14, i.e. with respect to the ground.
The drone 10 also comprises a measuring device 24 able to measure a ground speed Vsol_mes of the drone 10. The measurement device 24 is, for example, a satellite positioning device, also called a GNSS (Global Navigation Satellite System) device, or an inertial unit, also called IMU (Inertial Measurement Unit) with accelerometers and/or gyrometers, for measuring angular velocities and attitude angles of the drone 10.
In optional addition, the drone 10 comprises a pressure sensor (not shown), also called barometric sensor, configured for determining variations in the altitude of the drone 10, such as instantaneous variations and/or variations with respect to a reference level, i.e. with respect to a predefined initial altitude. The reference level is, for example, sea level, and the pressure sensor is then able to provide a measured altitude of the drone 10 with respect to sea level.
In addition, the drone 10 comprises a sensor (not shown) for measuring the air speed of the drone, this measurement sensor being connected to a dynamic pressure tap of a pitot probe type element. In optional addition, the drone 10 comprises a magnetometric sensor (not shown) giving the orientation of the drone with respect to the geographical north. The drone 10 is a motorized flying machine which is remotely controllable, in particular via a joystick 26.
The drone 10 comprises a transmission module 28 configured for exchanging data, preferably by radio waves, with one or more pieces of electronic equipment, in particular with the joystick 26, or even with other electronic elements for the transmission of the acquired image(s) by the image sensors 12, 18.
In the example of
Alternatively, the drone 10 is a rotary wing drone (not shown) which comprises at least one rotor, or preferably a plurality of rotors, the drone 10 being then called a multirotor drone. The number of rotors is, for example 4, and the drone 10 is then called a quadrotor drone.
The first image sensor 12 is known per se, and is, for example, a vertical-aiming camera pointing downwards.
The terrain 14 is understood in the general sense of the term, as a portion of the Earth's surface when it is an external terrain, whether it is a terrestrial surface or a maritime surface, or a surface comprising both a terrestrial portion and a maritime portion. In one variant, the terrain 14 is an interior terrain within a building. The terrain 14 is also called the ground.
The electronic control device 16 comprises a first acquisition module 40 configured for acquiring a succession of images of a terrain overflown by the drone 10, the images being taken by an image sensor fitted to the drone 10, such as the first image sensor 12, or even such as the second image sensor 18, wherein it should be understood that the images that are preferentially used for the application of the optical flow algorithm to the acquired images, are those provided by the first image sensor 12.
The electronic control device 16 comprises a second acquisition module 44 configured for acquiring the measured ground speed Vsol_mes, supplied by the measuring device 24 equipping the drone 10. The second acquisition module 44 is further configured for acquiring an altitude Zref_mes of the drone 10 measured with respect to a reference level. Optionally, the second acquisition module 44 is further configured for acquiring an altitude Zsol_mes of the drone 10 measured with respect to the terrain 14, i.e. with respect to the ground.
The electronic control device 16 comprises a calculation module 46 configured for calculating an altitude Zsol_est of the drone 10 with respect to the terrain 14, based on the acquired measured ground speed Vsol_mes, and an optical flow algorithm applied to the acquired images.
According to the invention, the electronic control device 16 further comprises a recalibration module 48 configured for correlating the altitude calculated with respect to the terrain Zsol_est and the altitude measured with respect to the reference level Zref_mes.
Optionally, the electronic control device 16 further comprises a control module 50 configured for controlling an attitude of the drone 10 as a function of an altitude of the drone 10, the control module 50 being configured for calculating control instructions corresponding to said attitude.
In the example of
The joystick 26 is known per se, and allows controlling the drone 10. In the example of
The controller 26 also includes a radio antenna 74 and a radio transceiver (not shown) for exchanging radio wave data with the drone 10, both uplink and downlink.
In the example of
Alternatively (not shown), the first acquisition module 40, the second acquisition module 44, the calculation module 46 and the recalibration module 48, as well as the optional control module 50, are each made in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array), or in the form of a dedicated integrated circuit, such as an ASIC (Applications Specific Integrated Circuit).
The first acquisition module 40 is further configured for calculating a first indicator Ind1 as a function of a gradient of each acquired image, and the calculation module 46 is then further configured for calculating, only when the value of the first indicator Ind1 is greater than a first threshold S1, the altitude Zsol_est of the drone 10 with respect to the terrain 14 by application of the optical flow algorithm to the acquired images.
The first acquisition module 40 is, for example, configured for calculating for each pixel, a gradient of intensity value between the intensity value of the pixel in question and that of the neighboring pixels, the intensity value of each pixel being, for example, expressed in gray level, for example in 8 bits with values lying between 0 and 255.
The first indicator Ind1 is then, for example, the number of pixels of the image for which the calculated gradient is greater than a predefined minimum gradient. The first threshold S1 is then, for example, 4, and the image is then considered to be sufficiently good for the application of the optical flow algorithm from the moment when the value of the gradient is greater than or equal to the predefined minimum gradient for at least 4 pixels of the said image.
The calculation module 46 is configured for calculating the altitude Zsol_est of the drone 10 with respect to the terrain 14 based on the optical flow algorithm applied to the acquired images and the measured ground speed Vsol_mes acquired by the second acquisition module 44.
The optical flow algorithm is known per se, and is generally used to estimate a ground speed from a predefined altitude of the drone with respect to the terrain, this predefined altitude being assumed to be substantially constant.
The optical flow algorithm makes it possible to estimate the differential movement of a scene from one image to the next image, and there are various known methods for implementing the optical flow algorithm, such as, for example, the Lucas-Kanade method, the Horn-Schunk method, or the Farneback method. The optical flow algorithm is furthermore capable of being implemented via a so-called multi-resolution technique, which is configured for estimating the optical flow with different successive image resolutions, starting from a low resolution to a high resolution.
The optical flow algorithm is also capable of being combined with another image processing algorithm, in particular with a corner detection algorithm, to improve the estimation of the differential movement of the scene from one image to the next, as described in EP 2 400 460 A1. Other examples of the implementation of an optical flow algorithm are also described in the documents “Optic-Flow Based Control of a 46 g Quadrotor” by Briod et al, “Optical Flow Based Velocity Estimation for Vision Based Navigation of Aircraft” by Julin et al, and “Distance and velocity estimation using optical flow from a monocular camera” by Ho et al.
The calculation module 46 is configured for implementing this optical flow algorithm in an inverse manner, by assuming the known ground speed and then seeking to calculate the value of the altitude Zsol_est of the drone 10 with respect to the terrain 14. The calculation module 46 is, in particular, configured for using for this purpose, as the predefined value of the ground speed, the value of the measured ground speed Vsol_mes, acquired by the second acquisition module 44.
The recalibration module 48 is configured for correlating the altitude calculated with respect to the terrain Zsol_est and the altitude measured with respect to the reference level Zref_mes, in order to have a more reliable value of the altitude of the drone 10 with respect to the terrain 14.
Optionally, the recalibration module 48 is configured, in particular, for estimating a current altitude Zsol_est(N) with respect to the terrain 14 from a current altitude Zsol_est(N) measured with respect to the reference level and a previous altitude Zsol_est(N−1) calculated with respect to terrain that has been correlated with a previous altitude Zsol_est(N−1) measured with respect to the reference level.
The person skilled in the art will understand that N is an integer index with a value greater than or equal to 1, designating the current value of the quantity in question, while the index N−1 then designates the previous value of the quantity in question corresponding to the last correlation performed, while the index 0 designates an initial value of the quantity in question.
The recalibration module 48 is preferably configured for estimating the current altitude with respect to the terrain Zsol_est(N) from the current altitude measured with respect to the reference level Zref_mes(N) and from the previous altitude calculated with respect to the field Zsol_est(N−1) which has been correlated with the previous measured altitude with respect to the reference level Zref_mes(N−1) in the event of an—at least temporary—failure of the calculation, based on the optical flow algorithm, of the altitude with respect to the terrain.
Optionally, the recalibration module 48 is further configured for calculating a second indicator Ind2 inversely proportional to the first indicator Ind1, and for correlating the calculated altitude with respect to the terrain Zsol_est, the measured altitude being compared to the reference level Zref_mes only when the value of the second indicator Ind2 is less than a second threshold S2. The value of the second threshold S2 is more restrictive than the value of the first threshold S1.
The control module 50 is configured for controlling the attitude of the drone 10. When the altitude measured with respect to the terrain Zsol_est is greater than a first predefined threshold altitude Z1, the control module 50 is configured for controlling the attitude of the drone 10 as a function of the altitude measured with respect to the terrain Zsol_mes, acquired by the second acquisition module 44, and preferably only as a function of this altitude Zsol_mes among the various altitudes mentioned above. The first predefined threshold altitude Z1 is, for example, substantially equal to 15 m.
When the value of the altitude measured with respect to the terrain Zsol_mes is lower than the first predefined threshold altitude Z1, the control module 50 is configured for controlling the attitude of the drone 10 in addition to the altitude with respect to the terrain Zsol_est, which is calculated by the calculation module 46. In other words, when the value of the altitude measured with respect to the terrain Zsol_mes is lower than the first predefined threshold altitude Z1, the control module 50 is configured for controlling the attitude of the drone 10 as a function of the altitude measured with respect to the terrain Zsol_mes and the altitude calculated with respect to the terrain Zsol_est, the calculated altitude being preferably the recalibrated altitude provided at the output of the recalibration module 48.
When the value of the altitude measured with respect to the terrain Zsol_mes is lower than a second predefined threshold altitude Z2, the control module 50 is configured for controlling the pitch of the drone 10 to a value greater than a predefined minimum landing pitch. The second predefined threshold altitude Z2 is for example substantially equal to 1.2 m.
In other words, when the value of the altitude measured with respect to the terrain Zsol_mes is lower than this second predefined threshold altitude Z2 which corresponds to an altitude close to the ground, the control module 50 is configured for giving the drone 10 an emergency pitch in the event that proximity to the ground was not previously detected, and where the value of the pitch of the drone 10 was not already greater than the predefined minimum landing pitch.
The operation of the drone 10 is, in particular the electronic control device 16 according to the invention, is now explained with the help of
During an initial step 100, different successive images of the terrain 14 overflown by the drone 10 are acquired by the first acquisition module 40, preferably from the first image sensor 12, such as a vertical-aiming camera pointing downwards.
Optionally, the first acquisition module 44 calculates the first indicator Ind1 relating to these different acquired images, and which is an indicator of the quality of the images acquired. The control device 16 then tests, in the next step 110, the value of the first indicator Ind1 with respect to the first threshold S1, i.e. it compares the value of the first indicator Ind1 with that of the first threshold S1.
In parallel, during step 120, a value of the measured ground speed Vsol_mes is acquired by the second acquisition module 44 from the measuring device 24, this measuring device 24 being, for example, a satellite positioning device, also called a GNSS device, such as a GPS receiver or GLONASS receiver, or an inertial unit, also called IMU.
If, during step 110, the test with respect to the first threshold S1 is positive, i.e. if the value of the first indicator Ind1 is greater than or equal to the first threshold S1, then the control device 16 passes to step 130 in which the calculation module 46 calculates the altitude Zsol_est of the drone 10 with respect to the terrain 14, by application of the optical flow algorithm to the acquired images, and from the value of the measured ground speed Zsol_mes.
Otherwise, if the test performed in step 110 is negative, i.e. if the value of the first indicator Ind1 is lower than the first threshold S1, then the control device 16 returns to step 100 to acquire new images of the terrain 14 overflown by the drone 10.
The optical flow algorithm used in step 130 is, for example, an optical flow algorithm using the Lucas-Kanade method.
In the next step 140, the calculation module 46 calculates the second indicator Ind2 which is inversely proportional to the first indicator Ind1, and tests this second indicator Ind2 with respect to the second threshold S2, i.e. compares this second indicator Ind2 with the second threshold S2.
In parallel, during step 150, a value of the measured altitude with respect to the reference level Zref_mes is acquired by the second acquisition module 44, for example from the pressure sensor, the reference level being for example sea level.
If, during step 140, the test with respect to the second threshold S2 is positive, i.e. if the value of the second indicator Ind2 is less than or equal to the second threshold S2, then the calculation module 46 transmits to the recalibration module 48 the value of the calculated altitude of the drone with respect to the terrain Zsol_est, and the control device 16 passes to step 160 during which the recalibration module 48 correlates the calculated altitude with respect to the terrain Zsol_est with the measured altitude with respect to the reference level Zref_mes.
Otherwise, if the test performed in step 140 is negative, i.e. if the value of the second indicator Ind2 is greater than the second threshold S2, then the control device 16 returns to step 100 to acquire new images of the terrain 14 overflown by the drone 10.
In step 160, the recalibration module 48 estimates, in particular, the current altitude Zsol_est(N) with respect to the terrain 14 from the current altitude Zref_mes(N) measured with respect to the reference level and the previous altitude Zsol_est(N−1) calculated with respect to terrain that has been correlated with the previous altitude Zref_mes(N−1) measured with respect to the reference level. This is particularly useful in case of at least a temporary failure of the calculation of the altitude with respect to the terrain Zsol_est based on the optical flow algorithm, i.e. in the case where one of the two tests with respect to the first and second thresholds S1, S2, described above, is negative.
In parallel, during step 170, a value of the altitude measured with respect to the terrain Zsol_mes is acquired by the second acquisition module 44 from the altimeter 20, such as a radio altimeter or an ultrasound range finder.
The control device 16 then proceeds to step 180 during which the control module 50 controls the attitude of the drone 10, in particular as a function of the altitude of the drone 10 with respect to the terrain 14. The control module 50 then calculates control instructions corresponding to the said attitude as a function of the said altitude of the drone with respect to the terrain 14, these control instructions being in particular intended for servomechanisms orienting the control surfaces 39.
For this purpose, the control module 50 is able to use the value of the altitude with respect to the terrain Zsol_est calculated by the calculation module 46, and preferably the recalibrated altitude provided at the output of the recalibration module 48, and/or the value of the altitude measured with respect to the terrain Zsol_mes and acquired by the second acquisition module 44, as represented in
More precisely, when the altitude measured with respect to the terrain Zsol_mes is greater than the first predefined threshold altitude Z1, the control module 50 controls, during the step 180, the attitude of the drone 10 as a function of the measured altitude with respect to the terrain Zsol_mes, and preferably only according to this altitude measured with respect to the terrain Zsol_mes among the different altitudes, measured or calculated, for the drone 10.
During step 180, when the value of the altitude measured with respect to the terrain Zsol_mes is lower than the first predefined threshold altitude Z1, i.e. when the drone 10 is soon likely to start its landing phase, or when the drone 10 has received a landing instruction, for example from the joystick 26, the control module 50 controls the attitude of the drone 10 as a function of the altitude measured with respect to the terrain Zsol_mes and the altitude calculated with respect to the terrain Zsol_est, the calculated altitude being preferably the recalibrated altitude provided at the output of the recalibration module 48.
Optionally, when the value of the altitude measured with respect to the terrain Zsol_mes is lower than the second predefined threshold altitude Z2, the control module 50 controls, during the step 180, the pitch of the drone 10 to a minimum predefined landing pitch, in order to implement an emergency pitch in the event that pitching of the drone 10 was not previously instructed.
At the end of the step 180, the control device 16 returns to step 100 to acquire new images of the terrain 14 overflown by the drone 10.
The electronic control device 16 according to the invention then allows recalibrating the altitude of the drone calculated with respect to the terrain Zsol_est, by correlating this altitude calculated with respect to the terrain Zsol_est and the altitude measured with respect to the reference level Zsol_mes, which makes it possible to have a more reliable value of the altitude of the drone 10 with respect to the terrain 14.
This is particularly effective in case of an—at least temporary—failure of the calculation, based on the optical flow algorithm, of the altitude with respect to the terrain, and the comparison of the first and second indicators Ind1, Ind2 with the first and second thresholds S1, S2 respectively, then allows effectively detecting such a failure of the calculation of the altitude from the optical flow algorithm.
Such a failure is, for example, likely to occur when the terrain 14 overflown by the drone 10 generates a scene varying slightly from one image to another, which then generates a relatively high calculation uncertainty upon applying the optical flow algorithm to acquired images.
The altitude with respect to the terrain thus calculated and recalibrated, is then more reliable, and is particularly useful during the landing phase, in order to better predict the approach to the ground and to anticipate the time when the control device 16 has to command a pitching up of the drone 10 in order to touch the ground.
The possible need to force the pitch of the drone 10 to a predefined minimum landing pitch, when the value of the altitude measured by the altimeter 20 is less than the second predefined threshold altitude Z2, further allows providing an emergency corrective procedure for landing the drone 10, especially in the event that the recalibrated altitude provided at the output of the recalibration module 48 is temporarily disturbed.
It is then conceivable that the electronic control device 16 and the control method according to the invention allow calculating more reliably the altitude of the drone 10 with respect to the terrain overflown, and then reducing possible jerkiness during the operation of the drone 10, especially during the landing phase.
| Number | Date | Country | Kind |
|---|---|---|---|
| 16 62264 | Dec 2016 | FR | national |
