Patent Grant
9377852
The present disclosure generally relates to the field of flight display applications and more particularly to systems and methods that apply eye tracking technology to flight displays and management systems.
Existing systems in a flight deck of an aircraft may use several types of user interface control devices. These control devices may require several steps in order to complete a task, which may require significant time and effort for a user.
Some systems in a flight deck may incorporate voice recognition to reduce workload. However, voice recognition systems may not provide accuracy that is sufficiently high for use in certain applications.
Therefore, there exists a need for improved systems and methods for receiving and providing information via an interface on a flight deck.
The present disclosure is directed to a method for managing a user interface. The method may include the step of detecting a gaze of a user within a display. The method also includes the step of correlating the gaze of the user to an item displayed on the display. A further step of the method entails receiving an input from the user related to the item.
The present disclosure is also directed to a user interface system. The user interface system includes an eye tracking sensor configured to detect a gaze of a user viewing a display. The user interface system also includes a processor in communication with the eye tracking sensor. The processor is configured to receive the gaze of the user and to correlate the gaze to an item displayed on the display. The user interface system also includes an input device in communication with the processor, the input device configured to receive an input from the user related to the item.
The present disclosure is also directed to an input processing method. The method includes the step of receiving a voice command from a user. The method also includes the step of detecting a gaze of the user within a display. The method also includes the step of correlating the gaze of the user to an item displayed on the display. A further step of the method is to confirm the voice command when the item corresponds to the voice command.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not necessarily restrictive of the present disclosure. The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate subject matter of the disclosure. Together, the descriptions and the drawings serve to explain the principles of the disclosure.
The numerous advantages of the disclosure may be better understood by those skilled in the art by reference to the accompanying figures in which:
Conventional flight decks include several types of user interface control devices. These devices may require the following steps in order to complete a task:
The tasks may be completed using any number of conventional input devices, such as a trackball, force sensitive device, displacement joystick, or a mouse. The resulting workload on a user may be substantial.
The use of a touchscreen may help eliminate some of the steps. For instance, a touchscreen interface may eliminate the step of visually locating the cursor on the display screen because the finger is the cursor device in a touchscreen. However, the user may still have to complete all of the other steps.
One example where the workload may be substantial is when a user is manipulating a map and wants to zoom in our out, or when the user is working on a checklist. These tasks may require the following actions:
In this example, the interface requires multiple controls (cursor control device, range knob), further contributing to the user's workload.
Another tool that may be used to reduce workload within the flight deck is voice recognition technology. Some voice recognition leading algorithms use both context (what is the user doing) and statistical modeling to provide improved recognition rates. The effectiveness of voice recognition as a tool within the flight deck depends on the accuracy of the technology. Some voice recognition methods have had partial success, but in order to attain a tool capable of ‘natural language’, alternate voice recognition context generators are required.
Embodiments of the present disclosure relate to systems and methods for using eye tracking in the user interface on a flight deck. Referring generally to
An example implementation of the system 100 is provided in
The eye tracking sensor 102 may be suitable for implementation in an aircraft. The eye tracking sensor 102 may include a single sensor or a plurality of sensors. The eye tracking sensor 102 may include a non-contact system for measuring eye motion in one embodiment. In another embodiment, the eye tracking sensor 102 includes an attachment to the eye such as a contact lens for measuring the movement of the eye. In another embodiment, the eye tracking sensor 102 is incorporated within an existing helmet or mask system used by the user, or within a head up display. In another embodiment, the eye tracking sensor 102 includes a sensor placed around the eyes to measure movement. In another embodiment, the eye tracking sensor 102 may be partially or completely embedded within the display 112 or within other sub systems on the aircraft. The eye tracking sensor 102 may also be located adjacent to or near a display on the aircraft. In addition, the eye tracking sensor 102 may be configured to detect the user's gaze within a single display or within two or more different displays. It is understood that the type of eye tracking sensor 102 is not limited to the examples described and any suitable technology for detecting a gaze of a user may be used with the user interface system 100.
The eye tracking sensor 102 may include a processor in one embodiment. In one embodiment, the processor for the eye tracking sensor 102 be incorporated as part of a separate eye tracking system in communication with the processor 104. In another embodiment, the processor for the eye tracking sensor 102 may be part of the processor 104 of the user interface system 100.
The processor 104 may be in communication with the eye tracking sensor 102 and the input device 106 as shown in
The processor 104 may be further configured to adjust a location of the item 108 within the display 112, or to otherwise modify or change information related to the item 108. The change may be based on an input received through the input device 106, or another factor such as the type of application currently running on the display 112. For example, if the user is viewing a map and fixes his gaze 116 on a particular location on the map, the processor 104 may be configured to center the map on the location corresponding to the user's gaze. The processor 104 may be further configured to adjust information on the display 112 based on the input received through the input device 106. In this example, the user may use the input device 106 to zoom in or zoom out on the location of the user's gaze.
In another example, the user may be viewing a checklist or other pageable or scrollable list containing any type of data and the processor 104 may be configured to adjust the position of the list or an item on the list in the display 112. If the user is viewing the list and moves his gaze 116 towards the bottom of the list, the processor 106 may be configured to advance the checklist to move the item corresponding to the user's gaze to the top of the checklist. Similarly, the gaze 116 may be used to advance or move an item upwards or to the left or right in the checklist or any other type of pageable or scrollable list. An input from the user may be required in order to advance the checklist. The processor 106 may be further configured to complete items on the checklist based on input received from the user through the input device 106.
In another example, the user interface system 100 may be used to review and complete the Electronic Checklist for the aircraft. The processor 104 may be configured for running Electronic Checklist software and is further in communication with on board and off board systems for completing the Electronic Checklist. The user views the display 112 that shows the Electronic Checklist and the eye tracking sensor 102 detects the gaze of the user on the display 112. The user may scroll up and down the page by adjusting his gaze and as detected by the eye tracking sensor 102 and providing an input through the input device 106.
In another example, the user interface system 100 may be used to change data within the flight deck. For example, the user may target his gaze 116 to an indexable value such as heading, speed, or the like. The user may then use the input device 116 to increment/decrement the value. In one example this may be performed by rotating a thumbwheel of the input device 106. The user may also use a voice command to increment/decrement the value.
In one embodiment, the user interface system 100 may restrict automatic scrolling or other automatic cursor events based upon gaze 116 due to the possibility of problems resulting from the unintentional nature of eye movement. Similarly, a positive user action (for example, a user input through the input device 106) may be coupled with eye gaze actions in order to execute any change. In one embodiment, a positive user action may be required in order to execute any change. In another embodiment, a positive user action may be required to execute critical changes only. Whether a positive user action is required to execute a change is generally predetermined depending on the circumstances and the configuration of the user interface system 100. For example, in one embodiment a positive user action may be required in order to permit panning using the user interface system 100, as this may be considered a critical change. In another embodiment, panning may be permitted by the user interface system 100 without requiring a positive user action.
The processor 104 may include any suitable computer processor. The processor 104 may be implemented as a separate system on the flight deck, or incorporated within an existing system on the aircraft, such as the Flight Management System. The processor 104 may be in communication with other systems for the aircraft, including both on board and off board communication systems.
The user interface system 100 may be configured to operate with existing display systems for the aircraft. The display may include a display device for displaying graphical, video, photographic, and textual information to a user. For example, the display may be a screen, multi-function display, monitor, cathode ray tube, liquid crystal display, head up display, head down display, projector, plasma, flat panel or any other type of display suitable for a flight deck. Similarly, the display may include a single display or a plurality of displays located in the flight deck.
The input device 106 is configured to receive input from the user. The input may include an acknowledgement, a response, a request, or a change to information related to the item 108 corresponding to the user's gaze 116. The user may provide input to the system 100 via any suitable input device 106. For example, the user may provide input via tactile feedback or haptic technology, such as a button, a touch screen, a multi-touch surface, a pressure-triggered screen with a stylus, a keyboard, mouse, a haptic device incorporating a tactile sensor, or any other suitable device for receiving inputs from a user. The input device 106 may also include a voice recognition system or device. The input device 106 may include an existing input device 106 for the aircraft, or a new input device 106. The input device 106 may also combine different input device types, such as a haptic system and a voice recognition system, or multiple types of haptic systems or the like. The input device 106 may be located on the user controls allowing rapid access for the user.
The eye tracking sensor 102, processor 104, and input device 106 may be separate sub-systems or may be incorporated into a single system. Similarly, each of the eye tracking sensor 102, processor 104, and input device 106 may be incorporated within existing systems and sub systems for the aircraft.
In one embodiment, the user interface system 100 may be used with the Flight Management System of the aircraft. For example, the user may tab through pages of the Flight Management System by looking at specific fields within the Flight Management System. The user interface system 100 may also be incorporated as part of a goal-based Flight Management System. For example, the user interface system 100 may be useful in meeting the goal of reducing user workload. Similarly, the user interface system 100 may be useful in other goal-based Flight Management System applications. The user interface system 100 may also be configured as part of a user driven interface in one embodiment.
The user interface system 100 may be used on a variety of aircraft, such as civilian aircraft, military aircraft, helicopters, regional jets, business jets, and other aircraft. The user interface system 100 may also be useful in other vehicles or contexts where it is useful to interface with an operator or user via eye tracking.
The present disclosure is also directed to a method 500 for managing a user interface as shown in
In embodiments, the method 500 may include additional steps. For example, a further step of the method 500 may include highlighting the item on the display. Another step of the method 500 may include adjusting a position of the item on the display. For example, the item may include a location on a map, and adjusting a position of the item on the display may include centering the location on the map on a center of the display. In another example, the item may include a checklist and adjusting a position of the item on the display may include moving the checklist up or down, or paging through the checklist, or otherwise placing the checklist or a particular item on the checklist in a more convenient location.
The step of adjusting a position of the item on the display may occur automatically (for example, when using a mapping application, the display may automatically center on the location the user is viewing). The step of adjusting a position of the item on the display may also occur in response to a user input. For example, in a mapping application the user may provide input in order to zoom in or zoom out of a view on the map.
The method 500 may be useful by allowing a user to use gaze to drive a center point. For example, the user's gaze may be used to determine a center point on a map. Similarly, a user may use their gaze to adjust the location of the map, for example by looking at a side area of the map or a different location of the map, the map may pan or move in the direction of the gaze. In addition, the user's gaze may be used to determine a zoom center point when the user wishes to zoom in on the map. For example, the view of the map may center on the current gaze of the user. The user may then provide input (for example, through an input device) to increment or de-increment the zoom on the map.
The method 500 may also support the use of voice recognition by providing context to a voice command. In this embodiment, the step of receiving an input from the user related to the item 506 includes receiving a voice command from the user. A further step of the method 500 may include determining a meaning of the voice command based on the voice command and the item. For example, if a user wishes to increase the speed of the aircraft, he may look at the airspeed control display. Using the method 500, his gaze will be detected and correlated to an airspeed control item within the display, as shown in
The additional step of the method 500 of determining a meaning of the voice command based on at least the voice command and the item may be completed by a voice recognition system. Similarly, determining the meaning of the voice command may be based on additional factors.
In one embodiment, additional steps may be required to complete the task associated with the voice command. For example, an additional step of the method 500 may include receiving an input from the user related to the voice command. This input may be provided in order to confirm the voice command. Whether this additional step is required will depend on many factors, such as the type of voice command. For example, critical changes such as changes to the flight plan may require an additional confirmation step before they can be executed.
The present disclosure is also directed to an input processing method 600 shown in
The method 600 may be useful in fulfilling regulatory requirements that relate to the use of voice recognition in the context of avionics. For example, regulations may require a secondary confirmation of a critical command issued through a voice recognition system. The method 600 and user interface system 100 may be useful for providing this confirmation.
The systems and methods of the present disclosure may provide several advantages. First, the systems and methods of the present disclosure may reduce a user's workload by reducing the number of steps required to alter an item or make a change within an application on an aircraft system. The systems and methods of the present disclosure may also support simplification of aircraft interfaces by reducing the number of input devices and controls, including control systems that may be imperfect or difficult to use such as a mouse or a trackball cursor control device. The systems and methods may also facilitate reduced workloads when zooming in or out of a map by reducing the need for two control systems.
In addition, the use of voice recognition and eye tracking in some of the systems and methods of the present disclosure may support accuracy in voice recognition systems. Similarly, the use of voice recognition may be useful in providing a secondary confirmation step to perform a critical function. This may assist in compliance with regulatory requirements.
In the present disclosure, the methods disclosed may be implemented as sets of instructions or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the methods disclosed are examples of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the method can be rearranged while remaining within the disclosed subject matter. The accompanying method claims present elements of the various steps in a sample order, and are not necessarily meant to be limited to the specific order or hierarchy presented.
It is believed that the present disclosure and many of its attendant advantages will be understood by the foregoing description, and it will be apparent that various changes may be made in the form, construction, and arrangement of the components without departing from the disclosed subject matter or without sacrificing all of its material advantages. The form described is merely explanatory, and it is the intention of the following claims to encompass and include such changes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4028725 | Lewis | Jun 1977 | A |
| 4725959 | Nagata | Feb 1988 | A |
| 5818423 | Pugliese et al. | Oct 1998 | A |
| 5926790 | Wright | Jul 1999 | A |
| 5974384 | Yasuda | Oct 1999 | A |
| 6173192 | Clark | Jan 2001 | B1 |
| 6571166 | Johnson et al. | May 2003 | B1 |
| 7089108 | Merritt | Aug 2006 | B2 |
| 7415326 | Komer et al. | Aug 2008 | B2 |
| 7606715 | Krenz | Oct 2009 | B1 |
| 7809405 | Rand et al. | Oct 2010 | B1 |
| 7881832 | Komer et al. | Feb 2011 | B2 |
| 7912592 | Komer et al. | Mar 2011 | B2 |
| 7967439 | Shelhamer et al. | Jun 2011 | B2 |
| 8139025 | Krenz | Mar 2012 | B1 |
| 8234121 | Swearingen | Jul 2012 | B1 |
| 8311827 | Hernandez et al. | Nov 2012 | B2 |
| 8405610 | Cole | Mar 2013 | B1 |
| 8515763 | Dong et al. | Aug 2013 | B2 |
| 20030110028 | Bush | Jun 2003 | A1 |
| 20050203700 | Merritt | Sep 2005 | A1 |
| 20070288242 | Spengler et al. | Dec 2007 | A1 |
| 20080065275 | Vizzini | Mar 2008 | A1 |
| 20090303082 | Larson et al. | Dec 2009 | A1 |
| 20100030400 | Komer et al. | Feb 2010 | A1 |
| 20100240988 | Varga et al. | Sep 2010 | A1 |
| 20100295706 | Mathan et al. | Nov 2010 | A1 |
| 20120081236 | Best et al. | Apr 2012 | A1 |
| 20120272179 | Stafford | Oct 2012 | A1 |
| 20130109478 | Matsumaru et al. | May 2013 | A1 |
| 20130194164 | Sugden et al. | Aug 2013 | A1 |
| 20130214998 | Andes et al. | Aug 2013 | A1 |
| 20130215023 | Bourret et al. | Aug 2013 | A1 |
| 20130283213 | Guendelman | Oct 2013 | A1 |
| 20130293488 | Na et al. | Nov 2013 | A1 |
| 20130321265 | Bychkov et al. | Dec 2013 | A1 |
| 20140085342 | Shoemaker | Mar 2014 | A1 |
| 20140176744 | Horowitz | Jun 2014 | A1 |
| 20140191946 | Cho | Jul 2014 | A1 |
| 20140240217 | Lorenceau | Aug 2014 | A1 |
| 20140268054 | Olsson et al. | Sep 2014 | A1 |
| 20140350942 | Kady et al. | Nov 2014 | A1 |
| 20140372944 | Mulcahy et al. | Dec 2014 | A1 |
| 20140375680 | Ackerman | Dec 2014 | A1 |
| 20150042552 | Tsoref | Feb 2015 | A1 |
| Entry |
|---|
| U.S. Appl. No. 13/248,814, filed Sep. 9, 2011, Barber. |
