Patent Application
20220413603
A computing system, such as a head-mounted display device (HMD), may employ an eye tracking sensor as a user input mechanism. An eye tracking sensor can be used to determine a gaze direction of an eye of a user, which can be used to identify objects, such as user interface objects, in the determined gaze direction.
Examples are provided related to using eye tracking systems comprising multiplexed diffractive elements. One example provides a head-mounted display device comprising a see-through display system including a transparent combiner having an array of diffractive elements. The head-mounted display device also includes an eye tracking system comprising one or more light sources configured to direct light toward an eyebox of the see-through display system, and an eye tracking camera. The array of diffractive elements comprises a plurality of multiplexed diffractive elements configured to direct images of a respective plurality of different perspectives of the eyebox toward the eye tracking camera.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
As mentioned above, a computing system may utilize an eye-tracking system to sense a user's gaze direction as a user input modality. Based on eye-tracking sensor data and one or more anatomical models to account for such parameters as eye and head geometries, an eye-tracking system can project a gaze line that represents a gaze direction from each sensed eye. The computing system then can use the resulting gaze direction(s) to identify, for example, a displayed virtual object that the gaze line from each eye intersects. Further, in the case of a see-through head-mounted augmented reality (AR) display system, image data from an outward-facing image sensor calibrated to the eye-tracking system can be used to identify any real objects in the real-world scene intersected by the gaze lines. Eye tracking can be used as an input mechanism or used to augment another input such as user input commands made via speech, gesture, or button.
An eye tracking system may comprise one or more illumination sources and one or more cameras. The one or more illumination sources are configured to direct light (e.g., infrared light) toward the cornea of the eye to produce glints (reflections from the cornea) that are imaged by the one or more cameras. Image data from each eye tracking camera is analyzed to determine the location of retinal reflections, the location of glint from each illumination source, and a location of the pupil, which may be used to determine a gaze direction.
As mentioned above, some HMDs are configured as see-through augmented reality display devices. Such devices may place illumination sources and eye tracking cameras in the periphery of a field of view to avoid obstructing a user's view of a real-world background. However, such peripheral camera placement produces oblique images of the user's eyes, which may pose problems for detecting some gaze directions. Furthermore, facial features (e.g., eyelashes, cheeks, etc.) can occlude the view of the eye when imaged from large oblique angles. Similarly, light from peripherally located illumination sources may be occluded by such facial features. For example, an obliquely placed illumination source may illuminate eyebrows instead of producing eye glints in some instances.
Such issues with peripheral light source and camera placement can pose particular difficulties when designing lower profile HMDs that are designed to be worn closer to the face (e.g. a device having a sunglasses form factor). In such a device, display hardware may be moved closer to the face relative to larger form factor devices. However, this moves illumination sources and cameras closer to the face as well, which may result in larger oblique angles and worse occlusion from facial features. Additionally, camera performance may degrade at closer distances and larger oblique angles.
Accordingly, examples are disclosed that relate to using a transparent combiner comprising an array of diffractive elements with multiplexed diffractive elements to help address such problems with eye tracking cameras and light sources positioned at oblique angles to an eye. In some examples, the multiplexed diffractive elements are configured to direct images of different perspectives of an eyebox to the camera (a region of space corresponding to an intended location of a user's eye during device use). As the array of diffractive elements is located on the transparent combiner to be in positioned in front of a user's eye, the array may avoid the possible occlusion problems encountered by an obliquely positioned eye tracking camera, as when one perspective is occluded, another perspective may provide an unoccluded view of the eye. In other examples, multiplexed diffractive elements are configured to direct an image of the eye to a plurality of different locations on the image sensor of the eye tracking camera, such that the system may still track the eye if one image of the plurality of images moves partially or fully off the image sensor. While various examples disclosed herein are discussed in the context of a holographic optical element (HOE) comprising multiplexed holograms, it will be understood that any suitable array of diffractive elements comprising multiplexed diffractive elements may be utilized, such as HOEs, diffractive phase gratings, metasurfaces, geometric phase gratings/holograms, diffractive optical elements (DOEs), and surface relief gratings/holograms.
Examples are also disclosed that relate to using multiplexed diffractive elements to direct light from an illumination source toward an eyebox from different perspectives. As one such example, an HMD comprises a transparent combiner having an array of diffractive elements comprising multiplexed diffractive elements. The multiplexed diffractive elements are configured to form one or more virtual light sources from an illumination source, and to direct light from the one or more virtual light sources toward the eyebox. Instead of illuminating an eye from primary illumination sources located at oblique angles, the multiplexed diffractive elements may illuminate the eye with virtual light sources located at frontal perspectives, which may help avoid occlusion from facial features.
The term “transparent combiner” and the like as used herein represent an optical component configured to be positioned in front of the eye to allow a user to view both a real-world background and to provide a path between an eyebox and an eye tracking system component (e.g. a camera and/or illumination source) via the transparent combiner. In some examples, a transparent combiner may have some opacity, whether at select wavelengths or broadly across the visible spectrum, yet still permit a real-world background to be viewed. In some examples, a same transparent combiner may be used to deliver images for display and for eye tracking, while in other examples different transparent combiners may be used for eye tracking and for image display.
HMD device 100 also comprises an outward-facing camera system, depicted schematically at 106, which may comprise one or more of a depth camera system (e.g., time-of-flight camera, structured light camera, or stereo camera arrangement), an intensity camera (RGB, grayscale, and/or infrared), and/or other suitable imaging device. Imagery from outward-facing camera system 106 can be used to form a map of an environment, such as a depth map.
HMD device 100 further comprises an eye tracking system to determine a gaze direction of one or both eyes of a user. The eye tracking system comprises, for each eye, an eye tracking camera (illustrated schematically for a left eye at 108) and an illumination system (illustrated schematically for a left eye at 110), the illumination system comprising one or more light sources configured to form glints of light on a cornea of a user. A right eye tracking system may have a similar configuration. As described in more detail below, one or more of light from illumination system 110 to an eyebox of the HMD device 100, and images of the eyebox to camera 108, are diffracted by the multiplexed hologram of transparent combiner of see-through display 104a to form light rays extending between the illumination system 110 and/or the camera 108 having two or more different perspectives relative to an eye of a user wearing the HMD device 100.
HMD device 100 also comprises a controller 112 and a communication subsystem for communicating via a network with one or more remote computing systems 114. Controller 112 comprises, among other components, a logic subsystem and a storage subsystem that stores instructions executable by the logic subsystem to control the various functions of HMD device 100, including but not limited to the eye tracking functions described herein. HMD device 100 further may comprise an audio output device 116 comprising one or more speakers configured to output audio content to the user. In some examples, a speaker may be positioned near each ear. In other examples, HMD device 100 may connect to external speakers, such as ear buds or headphones.
Illumination source(s) 210 may comprise any suitable light source, such as an infrared light emitting diode (IR-LED) or laser. In some examples, each illumination source comprises a vertical-cavity surface-emitting laser (VCSEL).
As mentioned above, transparent combiner 214 is configured to provide an optical path between an eyebox of HMD device 200 and an image sensor 212 and/or illumination source 210. In various embodiments, the transparent combiner 214 may comprise a waveguide, a prism, or a transparent substrate that supports the HOE 216.
HOE 216 comprises multiplexed holograms 218 with angular and wavelength selectivity. Wavelength selectivity may provide multiplexed holograms that diffract IR light while being relatively insensitive to visible light, while angular sensitivity limits a range of incident light angles that are diffracted by the HOE. In some examples, the multiplexed holograms may comprise volumetric refractive index gratings, and may be formed using any suitable material. Example materials include light-sensitive self-developing photopolymer films, multicolor holographic recording films, holographic recording polymers, and transparent holographic ribbons.
As mentioned above, multiplexed holograms 218 may perform various functions, including directing different perspectives of an eyebox toward an image sensor, directing a same perspective of the eyebox to a plurality of different locations on an image sensor, and/or forming one or more virtual light sources from an illumination source to provide glint light from different perspectives for eye tracking. In some examples, images of different perspectives of an eyebox may be directed to overlapping areas on an image sensor. Thus, in some examples, the eye tracking system may comprise an optional trained machine learning function 220 to process image data. A machine learning function may facilitate the processing of image data capturing overlapping images of an eyebox from different perspectives, and may output information such as a probable identification of each imaged glint (e.g. a light source identification and perspective identification for each imaged glint), a probable identification of an imaged retinal reflection, or even a likely gaze direction. In some examples, a machine learning function also may be configured to analyze eye tracking image data even where multiple perspectives are not overlapping. Machine learning function 220 may comprise any suitable algorithm, such as a convolutional neural network (CNN) and/or deep neural network (DNN). Machine learning function 220 may be trained using a training set of image data and associated ground truth eye position data.
HMD device 200 further comprises a communication system 224 to communicate with one or more remote computing systems. HMD device 200 also comprises a computing device 228 comprising computing hardware such as storage and one or more logic devices. The computing device 228 may store instructions executable by the computing system to perform various functions, including but not limited to those described herein that relate to eye tracking. Example computing systems are described in more detail below.
In the depicted example, HOE 304 comprises optical power configured to provide collimated images to camera 314. In other examples, any other suitable optical power may be encoded in an HOE. Further, as mentioned above, the images of eyebox 308 may be incident on the image sensor of camera 314 in an overlapping arrangement. As such, image sensor data from camera 314 may be transmitted to a processor, which may, for example, input the image sensor data into a trained machine learning function to disambiguate the images by identifying imaged glints (e.g. by a probability that a detected glint is associated with a selected light source and selected perspective), and/or to predict a gaze direction.
The example shown in
In other examples comprising a waveguide combiner, the multiplexed holograms may not collimate light as the light is incoupled into the waveguide. In such an example, the multiplexed holograms and waveguide are configured to redirect the different perspective images with a same number of bounces within the waveguide. Likewise, in other examples, a reflective HOE may be used in combination with a waveguide to couple light into the waveguide for eye tracking.
Some HMDs may utilize a relatively small area image sensor. In some such devices, there may be a risk that an image of a glint or pupil may move beyond an edge of the image sensor as the eye moves, thereby impacting eye tracking performance. Thus, in such an HMD, a transparent combiner comprising multiplexed holograms may be configured to direct an image of the eyebox to two or more different locations on an image sensor. As one image moves beyond an edge of the image sensor, the other may move fully onto the image sensor, thereby preserving eye tracking performance.
In some examples, an HOE such as HOE 504 may be used to create a defocused or distorted copy of the eye at a shifted location on an image sensor. While this may appear to corrupt the image, due to redundancy in the image the use a defocused or distorted image may enable camera 514 (and eye tracking software) to extract more information about the eye. As one example, two copies of an image of an eye may be imaged at shifted locations on an image sensor (e.g. image sensor 514), but with each copy having a different focal distance. This may increase the “depth of focus” of the device and potentially improve overall eye tracking.
In some examples, eye tracking system 500 may expand eyebox 508 by imaging multiple points in the eyebox that are laterally shifted relative to each other. As shown schematically by rays 512a and 512b, the multiplexed holograms may direct two (or more) images corresponding to ray 512a, and two (or more) images corresponding to ray 512b, onto the image sensor. In this case, the two images corresponding to ray 512a and the two images corresponding to ray 512b may be imaged at overlapping locations on the image sensor. As discussed above, a trained machine learning function may be employed to process image data from the image sensor and determine a location of a pupil of eye 506, wherein the trained machine learning function may be trained using labeled image data comprising overlapping images corresponding to an eye imaged using HOE 504.
A holographic element comprising multiplexed holograms can also be used to form virtual light sources and illuminate an eyebox from different perspectives. This may allow the formation of a greater number of glints on a cornea using a lesser number of physical light sources, and may allow light for a selected glint location to be directed toward the eye from a plurality of different directions.
Continuing, method 800 further comprises, at 814, acquiring an eye tracking image via the light received from the multiplexed holograms at the eye tracking camera. In some examples, at 816, the images of different perspectives of the eyebox are received in an overlapping arrangement. Likewise, in some examples, at 818, images are received at different locations on the image sensor of the eye tracking camera.
Method 800 further comprises, at 820, determining a location of a pupil of an eye. In some examples, at 822, the method comprises inputting image data into a trained machine learning function to determine the location of the pupil, while in other examples geometric computational methods may be used. Method 800 further comprises, at 824, outputting eye tracking data. For example, the eye tracking data may be used to determine a gaze direction of the user, which may then be output to various computer applications and/or services that utilize gaze data.
Method 900 further comprises, at 914, forming images of a same perspective of the eyebox at first and second locations on an image sensor of the eye tracking camera. In some examples, at 916, the method comprises forming a plurality of images of each of a plurality of different perspectives of the eyebox. In such an example, the plurality of images of each perspective may be spatially separated, but may overlap with images of other perspectives.
Method 900 further comprises, at 920, determining the location of a pupil based on image data from the image sensor. In some examples, at 922, the method comprises inputting the image data from the image sensor into a trained machine learning function. At 924, the method comprises outputting the eye tracking data. For examples, the method may comprise outputting a pupil location to a service or software application to determine a gaze direction.
In some examples, at 1010, the light is incoupled into a waveguide. In some examples, the incoupling optical element is configured to have optical power, for example to collimate the light.
At 1012, method 1000 further comprises receiving the light at a holographic optical element comprising multiplexed holograms. The holographic optical element may be allocated on a transparent combiner, such as the waveguide of 1010, or another suitable optical structure (e.g. a prism or transparent substrate).
At 1016, the method comprises diffracting the light towards an eyebox using the multiplexed holograms, thereby forming virtual light sources to provide glint light to the eyebox from different perspectives. In some examples, at 1018, the multiplexed holograms comprise transmissive holograms. For example, transmissive holograms may be used to outcouple light from a waveguide. Likewise, in some examples, at 1020, the multiplexed holograms comprise reflective holograms, e.g., for use in a free-space arrangement. As mentioned above, in some examples, the multiplexed holograms may comprise optical power.
In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
Computing system 1100 includes a logic machine 1102 and a storage machine 1104. Computing system 1100 may optionally include a display subsystem 1106, input subsystem 1108, communication subsystem 1110, and/or other components not shown in
Logic machine 1102 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
Storage machine 1104 includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 1104 may be transformed—e.g., to hold different data.
Storage machine 1104 may include removable and/or built-in devices. Storage machine 1104 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage machine 1104 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
It will be appreciated that storage machine 1104 includes one or more physical devices. However, aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
Aspects of logic machine 1102 and storage machine 1104 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
When included, display subsystem 1106 may be used to present a visual representation of data held by storage machine 1104. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 1106 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 1106 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 1102 and/or storage machine 1104 in a shared enclosure (e.g., in HMD device 100), or such display devices may be peripheral display devices.
When included, input subsystem 1108 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker (e.g., eye tracking systems 300, 400, 500, 600, or 700), accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
When included, communication subsystem 1110 may be configured to communicatively couple computing system 1100 with one or more other computing devices. Communication subsystem 1110 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow computing system 1100 to send and/or receive messages to and/or from other devices via a network such as the Internet.
Another example provides a head-mounted display device comprising a see-through display system comprising a transparent combiner having an array of diffractive elements, and an eye tracking system comprising one or more light sources configured to direct light toward an eyebox of the see-through display system, and also comprising an eye tracking camera, wherein the array of diffractive elements comprises a plurality of multiplexed diffractive elements configured to direct images of a respective plurality of different perspectives of the eyebox toward the eye tracking camera. In some such examples, the array of diffractive elements comprises optical power, and the plurality of multiplexed diffractive elements each is configured to collimate a respective image. Additionally or alternatively, in some examples the transparent combiner comprises a waveguide configured to direct incoupled light toward the eye tracking camera, and the multiplexed diffractive elements comprise transmissive diffractive elements configured to incouple image light into the waveguide. Additionally or alternatively, in some examples the multiplexed diffractive elements comprise reflective diffractive elements. Additionally or alternatively, in some examples an angular selectivity of one or more of the multiplexed diffractive elements varies across the eyebox. Additionally or alternatively, in some examples the images of the respective plurality of different perspectives are incident on an image sensor of the eye tracking camera in an overlapping arrangement. Additionally or alternatively, in some examples the head-mounted display device further comprises a logic machine and a storage machine storing instructions executable by the logic machine to receive image data acquired by the eye tracking camera, and input the image data into a trained machine learning function. Additionally or alternatively, in some examples the array of diffractive elements is further configured to direct an image to a plurality of locations on an image sensor of the eye tracking camera. Additionally or alternatively, in some examples the multiplexed diffractive elements further comprise one or more diffractive elements configured to form virtual light sources from the one or more light sources, a number of virtual light sources being greater than a number of light sources in the one or more light sources. Additionally or alternatively, in some examples the head-mounted display device further comprises a waveguide configured to direct light from the one or more light sources to the multiplexed diffractive elements.
Another example provides a head-mounted display device comprising a see-through display system comprising a transparent combiner, the transparent combiner comprising an array of diffractive elements, and an eye tracking system comprising an eye tracking camera configured to receive one or more images of an eyebox of the see-through display system, and the eye tracking system also comprising a light source configured to output light toward the array of diffractive elements, wherein the array of diffractive elements comprises a plurality of multiplexed diffractive elements configured to direct the light from the light source toward the eyebox from a respective plurality of different perspectives. In some such examples the plurality of multiplexed diffractive elements comprise reflective diffractive elements. Additionally or alternatively, in some examples the head-mounted display device further comprises a waveguide integrated with the transparent combiner, the waveguide configured to transmit light from the light source to the plurality of multiplexed diffractive elements, and the plurality of multiplexed diffractive elements comprise transmissive diffractive elements. Additionally or alternatively, in some examples the light source comprises one or more lasers. Additionally or alternatively, in some examples the one or more lasers comprises one or more vertical-cavity surface-emitting lasers.
Another example provides a head-mounted display device comprising a see-through display system comprising a transparent combiner and a waveguide, and an eye tracking system comprising one or more light sources configured to direct light toward an eyebox of the see-through display system, an eye tracking camera comprising an image sensor, and an array of diffractive elements included on the transparent combiner, the array of diffractive elements comprising a plurality of multiplexed diffractive elements configured to direct images of a first perspective of the eyebox to a plurality of spatially separated locations on the image sensor of the eye tracking camera. In some such examples the plurality of multiplexed diffractive elements comprise an angular selectivity of 15° or less. Additionally or alternatively, in some examples the plurality of multiplexed diffractive elements are further configured to direct a plurality of images of a second perspective of the eyebox to the image sensor. Additionally or alternatively, in some examples the array of diffractive elements is a transmissive array of diffractive elements. Additionally or alternatively, in some examples the array of diffractive elements is located on a waveguide.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
