Patent Application
20220413606
The present invent is related to an optical system with accurate eye-tracking function and a related method, more particularly, to an optical system which provides accurate eye-tracking in interactive virtual environments and a related method.
Virtual reality (VR) is an interactive computer-generated experience taking place within a simulated environment, that incorporates mainly auditory and visual, but also other types of sensory feedback like haptic. Augmented reality (AR) provides an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information. Mixed reality (MR) is the merging of real and virtual worlds to produce new environments and visualizations, where physical and digital objects co-exist and interact in real time. Most of existing VR/AR/MR applications are controlled by user hands using joysticks or touch screens, but the burden of carry these control devices may cause inconvenience. By incorporating eye-tracking capabilities into VR/AR/MR headsets, the user can use the eyes as an operational interface, wherein various visual elements can trigger certain responses and behaviors.
Eye tracking is a technology for acquiring eye positions and eye movement by measuring either the point of gaze or the motion of an eye relative to the head. Most information presented by a computer is provided to a user through visual information on a user interface presented on a display. However, the ability of the user to perceive visual information is limited. One such limitation is the area of active vision, which is small due to physiological reasons, e.g., the structure of an eye. Another limitation occurs in the displays themselves. For instance, mobile devices in particular have small screens (virtual field of view) that need to present a wide range of information.
Therefore, there is a need of an optical system and a related method for providing accurate eye-tracking in interactive virtual environments.
The present invention provides an optical system which provides accurate eye-tracking and includes an eye-tracking module and a head-mounted display. The eye-tracking module includes a first sensor module configured to capture one or multiple eye images of a user. The head-mounted display includes a processor configured to control an operation of the head-mounted display and provide a user interface, and a display configured to present the user interface. The user interface includes a virtual field of view and at least one UI element for receiving a gaze command associated with the one or multiple gaze points from the user. The at least one UI element is arranged within estimated field of view of the user and located outside a range of the virtual field of view of the user interface. The one or multiple gaze points of the user are computed based on the one or multiple eye images.
The present invention provides method of providing accurate eye-tracking. The method includes capturing one or multiple eye images of a user, computing one or multiple gaze points of the user based on the one or multiple eye images of the user, and providing a user interface. The user interface includes a virtual field of view and at least one UI element. The at least one UI element is arranged within an estimated field of view of the user and located outside a range of the virtual field of view of the user interface.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
In the embodiment illustrated in
In the embodiment illustrated in
In another embodiment of the optical systems 100 and 200, the processor 22 may be omitted. More specifically, the head-mounted display 10 and the eye tracker 20 may share the same processor 12 which is configured to control the operation of the head-mounted display 10 and the eye-tracker 20.
Step 310: capture one or multiple eye images of the user.
Step 320: compute one or multiple gaze points of the user based on the one or multiple eye images of the user.
Step 330: acquire the relationship between the field of view of the display 14 and an estimated field of view of the user.
Step 340: provide a virtual field of view and at least one UI element in the user interface 19 based on the one or multiple gaze points of the user and the relationship between the field of view of the display 14 and an estimated field of view of the user.
Step 350: receive one or multiple commands from the user.
Step 360: perform a corresponding action associated with the at least one user interface (UI) element when a received user command triggers the at least one UI element; execute step 310.
In the optical system 100, the sensor module 26 in the eye tracker 20 includes at least one image sensor (eye sensor) which is configured to capture one or multiple eye images of the user in step 310. The processor 22 in the eye tracker 20 is configured to receive the one or multiple eye images captured by the sensor module 26 and compute the one or multiple gaze points of the user in step 320. In addition, the processor 22 in the eye tracker 20 may further compute other eye-tracking related information based on the one or multiple eye images of the user, such as the confidence and the accuracy of estimated gaze point, the eyeball location in 3D coordinate and pupil-related information (e.g., size). The algorithms for eye-tracking operation may be implemented as, but not limited to, a process/software/firmware executed on the processor 22 of the eye tracker 20.
In the optical system 200, the sensor module 26 in the eye tracker 20 includes at least one image sensor (eye sensor) which is configured to capture the one or multiple eye images of the user in step 310. The processor 12 may receive the one or multiple eye images captured by the sensor module 26 of the eye-tracker 20 and compute the one or multiple gaze points of the user in step 320. In addition, the processor 12 may further compute other eye-tracking related information based on the one or multiple eye images of the user, such as the confidence and the accuracy of estimated gaze point, the eyeball location in 3D coordinate and pupil-related information (e.g., size). The algorithms for eye-tracking operation may be implemented as, but not limited to, a process/software/firmware executed on the processor 12 of the optical system 200.
In the optical systems 100 and 200, the sensor module 16 may include at least one scene sensor configured to capture the one or multiple images of the current field of view of the user, at least one audio sensor (such as a microphone) configured to receive the audio signal from the user, and/or at least one motion sensor (such as a gyro and/or an accelerometer) configured to detect the motion of the user (especially the head movement).
In the optical systems 100 and 200, the illumination module 24 may include one or multiple infrared light-emitting diodes (LEDs) for illuminating the eyes of the user in order to guarantee the necessary contrast between the iris and the pupil regardless of the eye color, particularly in a very dark or bright background, thereby increasing the accuracy of the sensor module 26 in the eye tracker 20 when registering the light reflected by the user eyes. However, the implementation of the illumination module 24 does not limit the scope of the present invention.
In the step 330, the processor 12 is configured to acquire the relationship between the field of view of the display 14 and the estimated field of view of the user. The estimated field of view of the user corresponds to the situation when the user looks straight ahead, and may be calculated based on the physical specifications of the optical system 100 or 200 and the data of average periorbital anthropometric measurements of the user. The actual field of view of the user may shift from the estimated field of view with the motion of the user's eye. The relationship between the field of view of the display 14 and the estimated field of view of the user includes at least the ratio in the size of the field of view of the display 14 to the size of the estimated field of view of the user, wherein the above-mentioned ratio is smaller than 1. The relationship between the field of view of the display 14 and the estimated field of view of the user may also include at least the position of the display 14 relative to the estimated field of view of the user, wherein the relative position may be estimated based on the physical specification of the head-mounted display 10 and the data of the average periorbital anthropometric measurements. The relative position may further be refined by the eye-tracking related information computed based on the one or multiple eye images of the user.
In the optical systems 100 and 200, the user interface 19 containing a virtual field of view and one or multiple UI elements UIE1-UIEN may be presented on the display 14, wherein N is a positive integer. In an embodiment, the UI elements UIE1-UIEN may be visual information presented on the display 14. In another embodiment, each of the UI elements UIE1-UIEN may be an abstract element which is invisible to the user. If a UI element is configured to interact with the user, it may further be connected with an event handler which is associated with a specific operation of the optical system 100 or 200 and is handled by the processor 12.
The UI elements UIE1-UIEN are configured to add interactivity to the user interface 19 and provide touch points for the user as they navigate their way around. Each UI element may be associated with, but not limited to, input control, navigation control, and information display. Each UI element includes a graphic element and a hit box. The graphic element determines the appearance of the UI element and may be associated with the function of the UI element. In an embodiment, the graphic element of a UI element maybe a checkbox, a radio button, a dropdown lists, a list box, a toggle button, toggles, or a date/time picker for input control. In another embodiment, the graphic element of a UI element may be a breadcrumb, a slider, a pagination, a slider, an icon, or an image carousel for navigation control In yet another embodiment, the graphic element of a UI element may be a tooltip, a progress bar, a notification, a message box, or a modal window for information display. However, the appearance of the graphic element in each UI element does not limit the scope of the present invention.
The hit box of an UI element is a virtual element which is invisible to the user and is connected to a corresponding event handler. When a gaze command associated with the one or multiple gaze points of the user triggers the hit box of a UI element, a predetermined action associated with the UI element maybe performed. When one or multiple trigger conditions are satisfied, the UI element is triggered by the gaze command. Exemplary trigger conditions include, but not limited to, when the user's gaze is located within the hit box and the user also sends a response from the I/O device or the sensors (i.e., microphone) to confirm the selection, the duration of user's gaze located within the hit box is longer than a specific fixation duration, the user's gaze is located within the hit box and the user sends a response in the form of a voluntary eye movement event (e.g., voluntary saccade, blink, etc.) to confirm the selection immediately or within a specific time interval, and the user's gaze trajectory crosses the response boundary of the hit box. However, the type of the trigger conditions associated with each UI element does not limit the scope of the present invention.
In the optical system 100 or 200, the I/O device 18 is configured to receive the command from the user. In an embodiment, the I/O device 18 may include any type of handed controller (such as a game pad or a game console) and/or any form of haptic device (such as a suit or a glove). The I/O device 18 is configured to detect and transfer the user's motion signal to the processor 12 of the optical system 100 or 200. In an embodiment, the processor 12 may control the operation of the optical system 100 or 200 based solely on the user command received by the I/O device 18. In another embodiment, the processor 12 may control the operation of the optical system 100 or 200 based on both the user commands received by the I/O device 18 and the gaze commands received by the UI elements UIE1-UIEN of the user interface 19.
In step 340, the processor 12 is configured to provide the virtual field of view and at least one UI element in the user interface 19 based on the one or multiple gaze points of the user and the relationship between the field of view of the display 14 and the estimated field of view of the user.
As depicted in
In step 350, the optical systems 100 and 200 may receive one or multiple commands from the user. As previously stated, a user command may be inputted via the I/O device 18, or a gaze command may be inputted via the UI elements UIE1-UIEN of the user interface 19. When the hit box of a specific UI element is triggered by a gaze command of the user when one or multiple predefined trigger conditions are satisfied, the processor 12 is configured to perform a predefined action. The above-mentioned predefined trigger conditions may include, but not limited to, user's gaze fixed longer than the fixation duration of the specific UI element, another button being pressed, a voice command being issued, intentional user blinks, or certain gaze path/pattern being detected. The above-mentioned predefined actions may include, but not limited to, content selection, previous, next, setting, close, back, home, show notification and lock screen.
In the present optical system, one or multiple UI elements are arranged outside the range of the virtual field of view of the user interface for providing gaze-based user interactions. The implementation of the UI elements can thus be more flexible and no longer limited by the small range of the virtual field of view of the user interface. In addition, more image data may be presented on the virtual field of view of the user interface. Therefore, the present invention can provides an optical system and a related method for providing accurate eye-tracking in interactive virtual environments.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
This application claims the benefit of U.S. Provisional Application No. 63/215,984, filed on Jun. 28, 2021. The content of the application is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63215984 | Jun 2021 | US |
