This application claims priority to foreign French patent application No. FR 1701342, filed on Dec. 21, 2017, the disclosure of which is incorporated by reference in its entirety.
The present invention relates to a method for dual harmonization of a head-worn display system for making the display of piloting information of an aircraft conform with the outside real world.
The invention lies in the technical field of the piloting human-system interface (HSI) for aircraft, such as, for example, helicopters or aeroplanes, equipped with a head-worn or helmet-mounted display system (HWD or HMD) and a head posture detection device DDP.
The head-up display systems, whether worn or not, make it possible to display in particular a “symbology” conforming to the outside world, that is to say a set of symbols whose position in front of the eye of the pilot allows for a superimposition with the corresponding elements in the outside world. It may be for example a speed vector, a target on the ground or in the air, a synthetic representation of the terrain or even a sensor image.
This conformal display requires knowledge of the position and the attitude of the aircraft and, for the head-worn display devices, the attitude of the display relative to a fixed reference frame linked to the aircraft. These various positions and attitudes are supplied by the avionics systems for those of the aircraft, and by the posture detection device DDP for those of the display.
For example and in particular, the avionics systems for supplying the position and attitude of an aircraft can be, respectively:
As is known, a harmonization of the head-worn display system is performed on installation of the display system, in a cockpit, in order to compute the corrections of angles to be made to switch from the display reference frame to the aircraft reference frame, and in order to obtain a conformal head-up display.
Now, some head-worn display devices these days have a certain mobility between the display device or display and the worn part of the posture detection system DDP, because of an absence of mechanical rigidity between these two elements, i.e. the display and the mobile worn part of the DDP, for example when there is a device for tilting the display alone outside of the field of view of the operator. There is then a need, when the display is once again tilted into the field of view of the operator, to once again proceed with a harmonization in order to compute new corrections of angle to be made to the head once the head-up display is installed and thus be able to display a conformal symbology in the display device worn on the head.
In order to make it possible and to facilitate this relatively frequent need for reharmonization, it is known practice to install a dedicated instrument on board the aircraft, called boresight reference unit or boresight reticle unit BRU.
The boresight reference unit BRU, installed in the cockpit facing the head of the operator displays a collimated symbol with an orientation that is fixed and known to the head-up system.
Each time there is a need to realign the conformal symbology, i.e. for reharmonization, the operator aligns a symbol displayed in his or her head-up display with the collimated symbol of the boresight reference unit BRU.
When the symbol displayed in the head-up display, i.e. the display is aligned on the collimated symbol, the detection-device output harmonization system then computes a rotation matrix from three correction angles, in order to reharmonize the attitude of the reference frame of the display relative to the reference frame of the aircraft.
The main fault with this harmonization system based on the use of a boresight reference unit BRU is the inclusion of an additional item of equipment dedicated to just this realignment or harmonization function and a cost in terms of installation complexity, an additional bulk and weight that can be restrictive, in particular for small civilian aircraft. This BRU equipment item has to be powered through electrical wiring and installed in a robust manner. This BRU equipment item requires a lengthy harmonization procedure when it is installed with an additional error entry. A risk of misalignment through movement is possible for example upon installation or during a maintenance operation.
Furthermore, the exact parameters of orientation of this boresight reference unit BRU on the bearer, i.e. the bearing structure of the aircraft, has to be also introduced into the helmet-mounted display system HMD, and the BRU unit has to then always remain perfectly fixed relative to the bearer. Now, the current mechanical technologies do not make it possible to guarantee a mounting of the BRU unit in the cockpit without a risk of variations over time. Indeed, the vibratory environment, the interventions of the pilot and of the maintenance operators in particular can provoke slight rotations or movements of the boresight reference unit BRU, which results in the introduction of an error on the line of sight that cannot be compensated and, in many cases, that cannot be detected, and therefore the prevention of any subsequent reharmonization.
A first technical problem is how to provide a head-worn display system and a harmonization method which makes it possible to realign the symbology on the outside world when the head-up display or viewing system HWD/HMD has a mechanism for releasing and re-engaging the display in the field of view of the pilot, a source of misalignment, and to avoid the use of a calibration landmark installed inside the cockpit, also a source of error.
A second technical problem is how to more accurately determine the relative orientation M01 between the display D0 and the mobile tracking element D2 of the head posture detection subsystem DDP when the head-up display system HWD/HMD has a mechanism for releasing and re-engaging the display in the field of view of the pilot.
A third technical problem is how to correct the orientation of the aircraft supplied by its inertial station relative to the Earth, in particular for the heading whose value is generally not known with sufficient accuracy for a conformal display.
To this end, the subject of the invention is a dual harmonization method for a head-worn display system for making the display of piloting information of an aircraft conform with the outside real world, the head-worn display system comprising: a transparent head-worn display D0; a head posture detection subsystem DDP having a mobile tracking first element D1 securely attached to the transparent display D0, a fixed second element D2 securely linked to the platform of the aircraft, and a means for measuring and determining the relative orientation M12 of the mobile tracking first element D1 relative to a reference frame of the fixed second element D2 linked to the platform; an attitude inertial device D3 for supplying the relative attitude M3t of the platform relative to a terrestrial reference frame linked to the Earth; a harmonization subsystem for the head-worn display system for making the display of piloting information on the display D0 conform with the outside real world, the harmonization subsystem having a dual harmonization computer and a human-system interface for managing and performing the implementation of the dual harmonization method.
The dual harmonization method is characterized in that it comprises the steps consisting in:
According to particular embodiments, the dual harmonization method comprises one or more of the following features, taken alone or in combination:
Another subject of the invention is a head-worn display system for making the display of piloting information of an aircraft on a display conform with the outside real world comprising: a transparent head-worn display D0; a head posture detection subsystem DDP having a mobile tracking first element D1 securely attached to the transparent display D0, a fixed second element D2 securely linked to the platform of the aircraft, and a means for measuring and determining the relative orientation M12 of the mobile tracking first element D1 relative to a reference frame of the fixed second element D2 linked to the platform; an attitude inertial device D3 for supplying the relative attitude M3t of the platform relative to a terrestrial reference frame linked to the Earth, securely fixed to the platform; a harmonization subsystem for the head-worn display system for making the display of piloting information on the display D0 conform with the outside real world, the harmonization subsystem having a dual harmonization computer and a human-system interface for managing and performing the implementation of the dual harmonization method.
The head-worn display system is characterized in that the harmonization subsystem is configured to: perform a series of an integer number N greater than or equal to 3 of different sightings Vi, i varying from 1 to N, performed through the display D0 by aligning a centred sighting visual pattern on any same fixed target of the outside real world, each sighting Vi corresponding to a different fixed position Pi of the centre of the sighting pattern on the display D0 and having a sighting vector {right arrow over (xi )} determined as a function of the position Pi, and, for each sighting Vi, acquiring the corresponding measurement {circumflex over (K)}l of the relative angular orientation of the tracking element relative to a DDP reference direction, that is fixed relative to the platform of the aircraft; then computing the matrix of relative orientation M01 between the display D0 in the tilted position and the tracking first element D1 as the right matrix {circumflex over (D)}, the solution of the system of equations:
According to particular embodiments, the head-up display system comprises one or more of the following features, taken alone or in combination:
The invention will be better understood on reading the following description of several embodiments, given purely by way of example and by referring to the drawings in which:
According to
The dual harmonization computer 36 can be an electronic computer dedicated specifically to the implementation of the dual harmonization method or a more general-purpose electronic computer provided to also implement other functions of the head-up display system 2.
Likewise, the human-system interface 38 can be a human-system interface dedicated only to performing the harmonization method or a more general human-system interface sharing other functions of the head-up display system 2.
The display system also comprises a means 42 for defining, measuring or determining the relative angular orientation M2t of the fixed second element 22 D2 relative to the Earth, and a means 44 making it possible to know the relative orientation M23 of the fixed second element 22 D2, linked to the platform 24, relative to the attitude inertial device 30 D3.
The means 44 is implemented in the form of a procedure performed on installation of the head-worn display system 2 and the orientation M23 is assumed constant over time.
The means 42 uses the data of the attitude inertial device D3, attached to the platform of the aircraft and configured to measure its own orientation M3t relative to the Earth, and the angular orientation M23 supplied by the means 44.
The conformal piloting information comprises, for example, a speed vector, a target on the ground, a synthetic representation of the terrain or even an image from an electromagnetic sensor, for example an infrared sensor.
It is noteworthy that, in the current state of the art of head-up display systems, the posture detection subsystem 16 DDP is relatively complex in practice because it implements two measurements:
Here, and according to a subsequently preferred embodiment, the posture detection subsystem 16 DDP is configured to supply raw DDP output data deriving as a priority from the direct optical measurements of the relative orientation between the tracking first element D1 relative to the fixed second element D2.
It is noteworthy also that, here, for simplification reasons, the platform and the attitude inertial device D3 are related. Generally, the means 44 for supplying the relative orientation M23 is configured to perform this function in two steps: a first step of transition by the platform in which the tri-axial reference frame of the attitude inertial device D3 is “aligned” on the tri-axial reference frame of the platform, then a second step in which the orientation of the fixed second element D2 is harmonized on the reference frame of the platform.
These comments have no impact here on the content of the present invention.
Subsequently, the means Mij making it possible to know the relative orientation of one reference frame “i” to another “j” are likened hereinbelow in this document to the matrix describing this orientation. Indeed, the orientation Mij of one reference frame relative to another can be described equally by:
Subsequently, the matrix Mij will be able to also be denoted M(i/j), the matrix Mij or M(i/j) describing the relative orientation of the reference frame “i” relative to “j” (or from “i” to “j”). If vi is the expression of a vector in the reference frame “i” and vj is the expression of this vector in the reference frame “j”, then the relationship applies. Consequently, there is the relationship: vi=M(i/j)*vj and the relationship of transition between reference frames: M(i/k) (from i to k)=M(j/k)*M(ij).
The basic principle of the harmonization method for the head-up display system according to the invention rests on the use of a predetermined element of the outside terrestrial landscape used as landmark and a certain number of sightings consisting in aligning or superimposing a reticle of the symbology, fixed with respect to the display, on this outside element according to a number of positions of the reticle on the display which depends on the degrees of freedom affected by error of the relative angular orientation of the display D0 with respect to the mobile tracking first element D1 of the posture detection subsystem secured to the head.
According to
In a first, launching step 204, a triggering of the procedure for harmonizing the display of information conforming with the outside terrestrial world is actuated by the user of the head-worn display system, for example by pressing and holding a button situated in the cockpit and dedicated to this realignment. The display system is then set in a harmonization mode.
Then, in a second step 206 of acquisitions of sighting measurements, a centred sighting visual patter, for example a reticle of the symbology, is set fixed on the display by the computer at different positions, for example the following three different positions: (P1) left of the display and vertically to the centre, (P2) right of the display and vertically to the centre, then (P3) horizontally to the centre and upwards. In the same second step 206, the corresponding sightings, respectively denoted V1, V2, V3, are performed by aligning or superimposing the reticle, placed at the different positions P1, P2, P3 on the display, on a predetermined element of the real outside terrestrial landscape serving as landmark. These sightings V1, V2, V3 must be performed by taking the head roll: once to the right about the roll axis, that is to say the sighting axis, once to the left about the roll axis. For an optimal performance, each position of the reticle can give rise to two sightings: head inclined to the left then to the right, but this condition is not necessary to perform a harmonization of quality.
For each sighting Vi, i varying from 1 to 3, a corresponding measurement matrix Ki, i varying from 1 to 3, of relative orientation of the mobile part D1 of the posture detection subsystem DDP relative to the device D2 forming the fixed part of the subsystem is measured by the posture detection subsystem DDP and computed by the subsystem itself or the electronic harmonization computer which is connected to it.
Then, in a third step 208, the harmonization computer solves the following dual harmonization equation: Ĝ·{circumflex over (K)}i·{circumflex over (D)}·{right arrow over (x)}i={right arrow over (y)}0, in which:
To solve the harmonization equation, the third step 208 uses the algorithms of the fifth and sixth embodiments described in the French patent application entitled “Global dual harmonization method and system for a posture detection system” and filed on the same date as the present French patent application, depending on whether the computation of the direction y0 is desired or not.
When the computation of the direction y0 is desired, the matrix G has to be known and the coordinates have to be expressed in the reference frame of G, the third step 208 uses the fifth embodiment of the dual harmonization algorithm described in the patent application entitled “Dual harmonization method for a posture detection subsystem incorporated in a head-worn display system” and implements a first set 212 of first, second, third computation substeps 214, 216, 218.
The fifth embodiment of the dual harmonization algorithm solves, through the first, second, third computation substeps 214, 216, 218, the system of equations: Ĝ·{circumflex over (K)}i·{circumflex over (D)}·{right arrow over (x)}i={right arrow over (y)}0 for i varying from 1 to 3.
In the first, initialization substep 214, a first series of right matrices {{circumflex over (D)}[s]} is initialized by setting {circumflex over (D)}[0] equal to I3, I3 denoting the identity matrix.
Then, the second, iterative substep 216 is repeated for passing from the iteration [s] to [s+1] by computing {right arrow over (y)}[s+1] then {circumflex over (D)}[s+1] using the following equations:
The series {{right arrow over (y)}[s]} and {{circumflex over (D)}[s]} converge respectively towards {right arrow over (y0)} and {circumflex over (D)}.
In the third, stopping substep 218, the iterative process performed through the second substep 216 is stopped when the limits {circumflex over (D)} and Ĝ are approximated with a sufficient accuracy.
It is noteworthy that the fifth configuration mode of the dual harmonization computation demands, as usage constraints, that the minimum number of measurements N is greater than or equal to 3 and that the vector family {{right arrow over (x)}i} is free. That means that, as a variant of the harmonization method of the dual harmonization method for the display described in
According to
In the fourth, launching step 304, identical to the first step 204, a triggering of the procedure 302 for harmonizing the display of information conforming with the outside terrestrial world is actuated by the user of the head-worn display system, for example by maintained pressure on a button situated in the cockpit and dedicated to this realignment. The display system is then set in a harmonization mode.
Then, in the fifth step 306 of acquisitions of sighting measurements, a reticle of the symbology is set fixed on the display by the harmonization computer at four different positions, for example the following four different positions: (P1) left of the display and vertically to the centre, (P2) right of the display and vertically to the centre, (P3) horizontally to the centre and upwards, then (P4) horizontally to the centre and downwards. In the same fifth step 206, the corresponding sightings, respectively denoted V1, V2, V3, V4, are performed by aligning or superimposing the reticle, set at the different positions P1, P2, P3, P4 on the display, on a predetermined element of the outside real terrestrial landscape serving as landmark. These sightings V1, V2, V3, V4 have to be performed by taking the head roll: once to the right about the roll axis, that is to say the sighting axis, once to the left about the roll axis. For optimal performance, each position of the reticle can give rise to two sightings: head tilted to the left then to the right, but this condition is not necessary to perform a harmonization of quality.
For each sighting Vi, i varying from 1 to 3, a matrix of corresponding measurement Ki, i varying from 1 to 4, of relative orientation of the mobile part D1 of the posture detection subsystem DDP relative to the device D2 forming the fixed part of the subsystem is measured by the posture detection subsystem DDP and computed by the subsystem itself or the electronic harmonization computer which is connected to it.
Then, in the sixth step 308, the harmonization computer solves the following dual harmonization system of equations: Ĝ·{circumflex over (K)}i·{circumflex over (D)}·{right arrow over (x)}i={right arrow over (y)}0, in which:
Here, the dual harmonization system of equations to be solved differs from that of the second embodiment of the harmonization method in that:
In this case, to solve the harmonization equation, the sixth step 308 uses the algorithm of the sixth configuration described in the French patent application entitled “Global dual harmonization method and system for a posture detection system” and filed on the same date as the present French patent application, and implements one of the second set 312 of fourth, fifth, sixth computation substeps 314, 316, 318.
The dual solving harmonization algorithm of the sixth configuration amounts to the solving of the system of equations: Ĝ·{circumflex over (K)}i·{circumflex over (D)}·{right arrow over (x)}i={right arrow over (y)}0 for i varying from 1 to 4, to the solving of the system of equations: {circumflex over (K)}i·{circumflex over (D)}·{right arrow over (x)}i={right arrow over (z)}0 for i varying from 1 to 4, by denoting {right arrow over (z)}0=ĜT·{right arrow over (y)}0.
In the fourth substep 314, a first series of right matrices {{circumflex over (D)}[s]} is initialized by setting {circumflex over (D)}[0] equal to I3, I3 denoting the identity matrix.
Then, the fifth, iterative substep 316 for passing from the iteration [s] to [s+1] is repeated by computing {right arrow over (z)}[s+1] then {circumflex over (D)}[s+1] using the following equations:
The series {{circumflex over (D)}[s]} converges towards {circumflex over (D)}.
In the sixth, stopping substep 318, the iterative process performed through the fifth substep 316 is stopped when the limit {circumflex over (D)} is approximated with a sufficient accuracy.
It is noteworthy that the dual harmonization algorithm of the sixth configuration demands, as usage constraints, that the minimum number of measurements N is greater than or equal to 4 and that the vector family {{right arrow over (x)}i} is free. That means that, as a variant of the dual harmonization method 302 for the display described in
Thus, the knowledge of the orientation of a BRU unit relative to the bearer has been replaced by the assumption of identity of the fiducial direction with different sightings. Thus, the rotation matrix M03 of orientation of the display relative to the fiducial reference frame D2, if it is unknown in the absolute, is identical in the different sightings. By using:
Advantageously, in addition to the saving of a calibration instrument such as the BRU and above all the complex installation thereof, the dual harmonization method according to the invention makes it possible to obtain a better alignment accuracy than that provided through the use of a BRU, particularly on the harmonization in terms of roll.
The dual harmonization method described above also makes it possible to dispense with the errors and drifts of relative orientation between a BRU and the inertial device D3.
Number | Date | Country | Kind |
---|---|---|---|
17 01342 | Dec 2017 | FR | national |
Number | Name | Date | Kind |
---|---|---|---|
5138555 | Albrecht | Aug 1992 | A |
6311129 | Lin | Oct 2001 | B1 |
6377401 | Bartlett | Apr 2002 | B1 |
6449559 | Lin | Sep 2002 | B2 |
7046215 | Bartlett | May 2006 | B1 |
7409290 | Lin | Aug 2008 | B2 |
8019538 | Soehren | Sep 2011 | B2 |
9541763 | Heberlein | Jan 2017 | B1 |
9785231 | Zimmerman | Oct 2017 | B1 |
9891705 | Lahr | Feb 2018 | B1 |
10216265 | Kirchner | Feb 2019 | B1 |
20070030174 | Randazzo | Feb 2007 | A1 |
20090138138 | Ferren | May 2009 | A1 |
20100156758 | Anders | Jun 2010 | A1 |
20110282130 | Krueger | Nov 2011 | A1 |
20120178054 | Jomander | Jul 2012 | A1 |
20140152792 | Krueger | Jun 2014 | A1 |
20150200449 | Sleight | Jul 2015 | A1 |
20150317838 | Foxlin | Nov 2015 | A1 |
20160012643 | Kezele | Jan 2016 | A1 |
20170148340 | Popa-Simil | May 2017 | A1 |
20170276508 | Hodge | Sep 2017 | A1 |
20170277199 | Hodge | Sep 2017 | A1 |
20170287342 | Hodge | Oct 2017 | A1 |
20170351325 | Hashimoto | Dec 2017 | A1 |
20180210627 | Woo | Jul 2018 | A1 |
20190041979 | Kirchner | Feb 2019 | A1 |
20190215505 | Ishii | Jul 2019 | A1 |
Number | Date | Country |
---|---|---|
2015165838 | Nov 2015 | WO |
Entry |
---|
Tuceryan, et al., “Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR”, Proceedings of the IEEE and ACM International Symposium of Augmented Reality, pp. 149-158, Munich, Germany, Oct. 5, 2000. |
Number | Date | Country | |
---|---|---|---|
20190196192 A1 | Jun 2019 | US |