The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein:
The present invention provides a helmet position measuring system and a helmet mounted eye-gaze direction sensing system, together with associated methods.
The principles and operation of systems and methods according to the present invention may be better understood with reference to the drawings and the accompanying description.
Referring now to the drawings,
Each of these subsystems and its particular novel features will be described separately below.
Two common considerations pervade preferred implementations of the various subsystems of the present invention. Firstly, each subsystem is preferably implemented either totally without integration into electronic systems of the platform, or at least minimizing any required integration as far as possible, as will be detailed below. This greatly simplifies the installation procedure and facilitates “retrofit” of the systems on existing platforms without requiring the same level of evaluation and testing as would be required for an integrated system.
The second consideration pervading preferred implementations of the various subsystems of the present invention is the desire to minimize the excess weight and bulk of the helmet so that the helmet remains as close as possible to the size and weight of a conventional “dumb” helmet. To this end, any components which do not need to be helmet-mounted are preferably mounted in a separate body-mounted unit 18 (
In a further related consideration, the safety of the user is preferably enhanced by use of low power electronic components so as to avoid high-power connections between the helmet and platform systems. According to one option, a power supply 19 may be a self-contained battery unit, thereby avoiding power-supply connection to the platform. More preferably, a simple power-jack connector is used to supply low-voltage power to the helmet system. A battery power supply 19 may optionally be used to back-up the external power connection.
The various subsystems of preferred implementations of the invention will now be described individually.
As mentioned earlier, helmet-mounted inertial tracking systems alone are insufficient to determine motion of a helmet relative to a non-inertial platform. To address this problem, the inertial tracking system of the present invention preferably provides an inertial measurement system which includes an inertial measurement unit 12 associated with the helmet, and a communication link (transceivers 20a and 20b) associated with both the helmet and the platform for conveying data from an inertial navigation system (“INS”) 500 of the platform to the helmet system. A processing system 22 associated with inertial measurement unit 12 and communication link 22a, processes data from inertial measurement unit 12 and from the inertial navigation system 500 to derive helmet position data indicative of the helmet position relative to the moving platform. Parenthetically, it should be noted that the term “helmet position” when used herein as a stand-alone term is used to refer to either or both of angular position (attitude) and linear spatial position (displacement). When referring to parameters of motion, the convention of “position”, “velocity” and “attitude” is used wherein “position” refers specifically to position in three-dimensional space relative to a set of reference coordinates.
In addition to the basic functionality of calculating differential motion, it is a further feature of particularly preferred implementations of the present invention that transfer alignment is used to “align” the reference axes of IMU 12 with the reference axes of INS 500, thereby enhancing the precision of the measurement, bringing the output of the small and relatively low-precision head-mounted system up to a precision close to that of the much more sophisticated platform INS. Transfer alignment is a well known technique, typically used for inertial measurement systems rigidly fixed, or at least tethered, to a common platform, for correcting one system on the basis of a more accurate system moving on the common platform. Transfer alignment has not heretofore been employed in a helmet tracking system and would conventionally be discounted as impossible since the helmet is essentially free to move with the head of the user relative to the platform. In practice, however, for a rapidly moving platform such as an aircraft, the present invention points out that the velocity of the helmet may be assumed for calculational purposes to be identical to that of the platform. Based upon this observation, the present invention teaches the use of transfer alignment for enhancing the precision of measurement. A further distinctive feature of preferred implementations of the transfer alignment of the present invention is that the moving platform NS motion data for performing the transfer alignment is transmitted to the helmet system wirelessly via the wireless communications link (transceivers 20a and 20b).
A preferred implementation of the inertial, or hybrid, helmet position subsystem is illustrated schematically in
In addition to the basic calculation module described thus far, the helmet motion data for velocity 208 and attitude 210, and the platform motion data for attitude 214, velocity 216 and position 218 are preferably fed to Kalman filter 222 which implements transfer alignment algorithm to generate corrections to increase accuracy of the inertial measurement unit output. Preferably, the corrections include sensor corrections 224a and 224b for correcting bias or other errors in the readings from the inertial sensors, and velocity and attitude corrections 226 which adjust the current output motion data parameters which also serve as the basis for the subsequent integrated motion data calculations. The implementation of the transfer alignment filter is essentially the same as is used conventionally in many “smart” weapon systems, and will not be discussed here in detail.
The connections 224a, 224b and 226 are typically updated at a rate limited primarily by the processing capabilities or by the quantity of data required for endive convergence of the transfer alignment calculations. A typical example for application of these corrections would be a rate of about 1 Hz.
As will be discussed further below, the helmet tracking system is preferably implemented as a hybrid system which includes additional helmet tracking subsystems, and most preferably, an optical helmet hacking system 14. In this case, Kalman filter 222 provides a highly effective tool for combining the available information from multiple sources, with differing refresh rates, and with self-adaptive relative weighting of the information sources. In the case of an optical subsystem which measures helmet attitude relative to the platform a preprocessing step is performed by filter 222 to transform the measurements by use of platform attitude data 214 into the LLLN frame within which the Kalman filter computation is performed.
As mentioned earlier, most preferred implementations of the present invention try to minimize integration of the subsystems with the platform electronics systems. In order to obtain the required data from the INS in a minimally integrated way, the communication link 22b is preferably a wireless communication link associated with a peripheral device which already has read-access to the INS data. In the preferred example illustrated here, communication link 22b is associated with a weapon interface and controller 24 which interface with a weapon system 502. Weapon system 502 is itself connected to a data bus 504 or equivalent dedicated wiring which makes available information from multiple systems of the platform, including from INS 500. Thus, weapon interface and controller 24 can access data from INS 500 without itself being directly integrated in the electronics systems of the platform.
By way of a practical example, in the case that weapon system 502 is an advanced missile system including one or more missile having its own internal INS, a data bus connection providing the missile system with aircraft INS data typically already exists in order to allow transfer alignment of the missile INS using the aircraft data as a reference. In this case, by tapping into the missile data directly, the data required by helmet system 10 may be retrieved without any modification of the aircraft hardware or software. The data connection may be achieved either through connection with a processing unit within the missile itself, or through connection with a processing unit within the missile launcher unit.
Turning now additionally to
Although the inertial helmet position sensing system described thus far is believed to be highly effective in its own right, most preferred implementations of the present invention employ a hybrid helmet tracking system with a second preferably optical subsystem providing corrective data. A preferred example of the optical helmet position subsystem will now be described with reference to
In order to provide an optical helmet position tracking system with minimal integration into systems of the platform, it is a particularly feature of most preferred implementations of the optical tracking system that the only “installed” elements outside the helmet system itself are passive reflectors 30, typically applied as stickers positioned within the cockpit or other working environment. At least three, and typically four, reflectors 30 are used, and they may have identical shapes and sizes, or may be geometrically distinct. The reflectors are preferably directional reflectors which reflect maximum intensity along a line roughly parallel with the incoming illumination.
In order to operate with passive reflectors 30, optical sensor arrangement 14 includes a helmet-mounted illumination system 32 for directing electromagnetic radiation of at least one wavelength from the helmet in at least one range of angles, and a helmet-mounted imaging system 34 sensitive to at least the at least one wavelength for deriving images of part of the predefined environment including electromagnetic radiation reflected from reflectors 30. Processing system 22 then processes the images to identify regions of the images corresponding to reflectors 30 and hence to determine information relating to a position of helmet 28 within the predefined environment.
Preferably, illumination system 32 includes at least one infrared LED, and most preferably two, three or four LED's which together cover substantially the entire field of view of imaging system 34. This preferably corresponds to a substantially continuous horizontal angular range of at least 60°, and a substantially continuous vertical angular range of at least 45°. In this context, the terms “horizontal” and “vertical” are used to refer to directions as perceived by the user in his or her normal orientation on the platform. Optionally, the optical system may be supplemented by one or more additional illumination system 32 and imaging system 34 mounted on the helmet with additional viewing directions in order to enlarge the range of angles over which reflectors 30 are within the FOV. Alternatively, an enlarged set of reflectors may be positioned to provide distinctive reflective symbols over an increased range of angles and/or in different viewing directions. For example, a secondary set of IR reflective stickers which are transparent to visible light may be deployed on a cockpit canopy to provide optical tacking when the user looks “up” in an aircraft frame of reference.
For reliable optical tracking, it is desired to achieve high contrast imaging of reflectors 30 while using low power illumination, despite the fiat that the system operates in an environment which may be exposed to direct solar radiation. Surprisingly, it has been found that these conditions can be met very successfully by employing directional reflectors (i.e., which return a majority of the reflected illumination intensity in a direction roughly parallel with the incoming illumination) in combination with narrow waveband wavelength selection. Thus, in most preferred implementations, at least the imaging system 34 is configured to be at least partially selective to electromagnetic radiation of a wavelength or wavelength band emitted by illumination system 32. This can be achieved most simply by positioning a suitable filter element 36 in front of at least the imaging sensor 34.
The calibration procedures and the processing required for position determination from the images obtained are known in the art and are typically similar to those of the commercially available systems mentioned earlier.
Each of the aforementioned helmet tracking subsystems has its own advantages and disadvantages. The inertial system offers large bandwidth (rapid response) and operates over effectively limited angular range, but may suffer from errors or “drift”, particularly under low-acceleration conditions where insufficient data may be available for effective transfer alignment. The optical system on the other hand, once calibrated, offers repeatable accuracy and zero drift, but suffers from relatively slow response (typically around 5 Hz) and limited angular range. The two systems therefore complement each other perfectly to provide a hybrid helmet tracking system which combines the advantages of both subsystems. A preferred structure for integrating the measurements of the different subsystems was described above with reference to
Turning now to the eye-tracking subsystem and associated method, a preferred structural layout of the eye-tracking optical components is illustrated in
The use of hot-mirror 64 enables LED 60 and camera 62 to be located in the peripheral region of helmet 28 near the edge of the visor. For extra compact, depending upon the size and shape of camera 62, it may be advantageous to employ an extra mirror 66 to allow mounting of the camera vertically or in any other preferred orientation.
The eye-tracking subsystem also includes processing and data storage components, as well as power supply and driver circuitry, as will be clear to one ordinarily skilled in the art. The processing and data storage components are typically included in the general designation of processing system 22 (
Turning now to the operation of the eye-tracking subsystem and the corresponding method, as mentioned earlier, there are two known techniques for deriving eye-gaze direction from images of the eye, referred to herein as “pupil-plus-glint” and “pupil only”. These individual techniques are known per se and are included in commercially available products as detailed in the background to the invention above.
To address this problem, it is a particularly preferred feature of the eye-tracking subsystem and corresponding method of the present invention that it combines the stability of the pupil-plus-glint tracking method with a range of tracking angles beyond the range which provides direct corneal reflection. This is achieved by using real-time automatic switching between two tracking calculation techniques, and most preferably, by automatic self-calibration of the pupil-only tracking technique based upon output of the pupil-plus-glint calculation technique during continuous operation of the system.
Turning specifically to
According to most preferred implementations of the present invention, values of the gaze direction derived from the pupil-plus-glint calculation are used to update at least one parameter of the pupil-only model (step 84). In the case of an algebraic formula, this is typically done by adjusting one or more coefficient of the formula. In the case of a look-up table, adjustment may be made either to individual values or by scaling a plurality of values.
By updating the pupil-only model frequently, or substantially continuously, it can be ensured that the pupil-only model is optimize for the current position of the helmet and working conditions, thereby substantially eliminating the cumulative sources of error normally associated with the pupil-only eye-tracking technique.
According to a further supplementary, or alternative, aspect of the present invention, it is possible to provide multiple illumination directions of the eye such that at least one direct corneal glint is received by camera 62 over an enlarged range of gaze direction angles. The additional illumination directions are most simply achieved by providing additional hot-mirrors 64 suitably angled and positioned across the inner surface of the visor, each with its own illumination source (LED 60). Although it is possible to use multiple cameras for each eye to achieve a similar result, the use of a single camera with multiple illumination directions is typically preferred for its reduced image processing load. The matching of each glint with the corresponding illumination direction is typically straightforward by use of the relative geometry of the pupil and glint positions in the images. According to one preferred alternative implementation, a total of three or more illumination directions are used to ensure a direct glint over substantially the entire range of angular motion of the eye, thereby rendering the use of the pupil-only mode unnecessary.
In the preferred case of binocular eye tracking, the two individual eye-gaze directions are correlated at step 86. In the case of assumed far-vision parallel binocular fixation, the two individual gaze directions may be assumed to be parallel and can be combined to improve output accuracy. Each measurement may be given equal weight, or an adaptive filter technique may be used to give variable weight depending upon different regions of greater or lesser measurement accuracy for each eye, or as a function of which calculation technique was used for each eye.
In the case of the preferred combination of features of the present invention, the eye-gaze direction relative to the helmet is then combined with helmet position data input at step 88 and the gaze-direction relative to the platform is calculated (step 90).
The helmet system described herein is useful for a wide range of different applications. In the specific version shown herein in the drawings, it is particularly useful as part of a system such as is described in the aforementioned co-assigned, co-pending U.S. Patent Application, published as Publication No. 20020039073 to provide a helmet-based cuing system without requiring a helmet mounted display. It should be noted, however, that any or all of the features of the present invention may equally be used to advantage in the context of a helmet which includes a helmet mounted display (HMD).
It should also be noted that the helmet system of the present invention, with or without a HMD, may also be used as a powerful tool for training or debriefing users. Specifically, it will be noted that preferred implementations of system 10 inherently generate helmet tracking information, eye tracking information and a forward-looking image from image system 34. By recording some or all of this data, optionally time-correlated to other actions of the user, databus information or external events, it is possible to reconstruct the movements of the user's head and his or her eye motion in the context of the forward view image. The data may either be recorded within data storage devices within processing system 22 or by a separate data storage unit (not shown) with a hard-wired or wireless one-directional communications link. The data storage device may optionally be part of an impact-protected disaster-investigation system.
By way of one non-limiting example, it is possible to replay the forward-view video images with the user's gaze direction superimposed thereon, thereby documenting the visual awareness of the user and the time-division of his or her attention. Optionally, the playback mode can simultaneously display flight information of the aircraft, as well as flight information of other aircraft or any other data or parameters available from the databus. The combined data can also be used to reconstruct the progression of events in three-dimensions.
It will be appreciated that the above descriptions are intended only to serve as examples, and that many other embodiments are possible within the scope of the present invention as defined in the appended claims.
Number | Date | Country | Kind |
---|---|---|---|
159061 | Nov 2003 | IL | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IL2004/001067 | 11/18/2004 | WO | 00 | 4/19/2007 |