Patent Grant
11042031
The disclosure generally relates to eye tracking technologies and particularly relates to eye tracking technologies for wearable heads-up displays.
A wearable heads-up display is a head-mounted display that enables the user to see displayed content but does not prevent the user from being able to see the external environment of the user. The display component of a wearable heads-up display is either transparent or at a periphery of the user's field of view so that it does not completely block the user from being able to see the external environment.
Eye tracking is a process by which one or more of position, orientation, and motion of an eye may be measured or monitored. In many applications, this is done with a view towards determining the gaze direction of a user. There are various techniques for measuring the position, orientation, and/or motion of the eye, the least invasive of which employs one or more optical sensors, e.g., cameras, to optically track the eye. Common techniques involve illuminating or flooding the entire eye, all at once, with infrared light and measuring reflections from the eye with at least one optical sensor that is tuned to be sensitive to the infrared light. Information about how the infrared light is reflected from the eye is analyzed to determine the position, orientation, and/or motion of one or more eye features such as the cornea, pupil, iris, and/or retinal blood vessels.
Eye tracking functionality is highly advantageous in wearable heads-up displays. Some examples of the utility of eye tracking in wearable heads-up displays include influencing where content is displayed in the user's field of view, conserving power by not displaying content that is outside of the user's field of view, influencing what content is displayed to the user, determining where the user is looking or gazing, determining whether the user is looking at displayed content on the display or through the display at the external environment, and providing a means through which the user may control or interact with displayed content.
However, incorporating eye tracking functionality in a wearable heads-up display conventionally adds unwanted bulk to the system. Eye tracking systems available today generally implement multiple dedicated components with very stringent positioning requirements, which undesirably increase the overall size and form factor of the system when incorporated into a wearable heads-up display. There is a need in the art for systems, devices, and methods of eye tracking that can integrate into wearable heads-up displays with minimal effect on the size and form factor of the system.
An eye tracking system for tracking an eye of a user may be summarized as including at least one infrared laser diode to output an infrared light, an optical scanner positioned to receive the infrared light outputted by the at least one infrared laser diode and controllable to scan the infrared light over a target area, a holographic optical element positioned at the target area to receive the infrared light scanned by the optical scanner and to redirect the infrared light in a direction of the eye of the user, an infrared detector aligned to detect at least a portion of the infrared light returned from the eye, and an infrared filter disposed in a position to selectively block transmission of infrared wavelengths through the holographic optical element from a side of the holographic optical element.
The infrared filter may have characteristics to block transmission of infrared wavelengths in a select range of infrared wavelengths to which the infrared detector is sensitive.
The infrared detector may be aligned to receive the at least a portion of the infrared light returned from the eye from the holographic optical element. An additional infrared filter may be disposed in a path through which the infrared detector receives infrared light from the holographic optical element, and the additional infrared filter may have characteristics to remove unwanted infrared wavelengths from the infrared light detected by the infrared detector.
The holographic optical element has a back side that faces the eye and a front side opposite to the back side that faces away from the eye, and the infrared filter is disposed on or proximate the front side. The infrared filter may be physically coupled to the front side of the holographic optical element.
The eye tracking system may include at least one transparent lens having a body, and the holographic optical element may be embedded in the body of the at least one transparent lens. The infrared filter may be embedded in the body of the at least one transparent lens or disposed on a surface of the body of the at least one transparent lens.
The holographic optical element may include at least one infrared hologram that is responsive to infrared light and unresponsive to visible light. The holographic optical element may further include at least one visible hologram that is responsive to visible light and unresponsive to infrared light.
The eye tracking system may include at least one visible laser diode to output a visible light. The optical scanner may be positioned to receive both the infrared light outputted by the at least one infrared laser diode and the visible light outputted by the at least one visible laser diode.
An eyeglass lens may be summarized as including a holographic optical element, an infrared filter disposed in a position to selectively block transmission of infrared wavelengths through the holographic optical element from a side of the holographic optical element, and a transparent lens having a body, wherein the holographic optical element and the infrared filter are physically coupled to the body.
The holographic optical element and the infrared filter may be embedded in the body of the transparent lens.
The holographic optical element may be embedded in the body of the transparent lens, and the infrared filter may be disposed on a surface of the body of the transparent lens.
The holographic optical element may include at least one infrared hologram that is responsive to infrared light and unresponsive to visible light. The holographic optical element may further include at least one visible hologram that is responsive to visible light and unresponsive to infrared light.
A wearable heads-up display having a field of view may be summarized as including a head wearable support frame that when worn positions the field of view with respect to at least one eye, at least one infrared laser diode coupled to the support frame and operable to output an infrared light, and at least one visible laser diode coupled to the support frame and operable to output a visible light. The wearable heads-up display may include at least one scan mirror coupled to the support frame and aligned to receive both the infrared light from the at least one infrared laser diode and the visible light from the at least one visible diode. The at least one scan mirror may controllably reflect both the infrared light and the visible light. The wearable heads-up display may include a holographic optical element coupled to the support frame and aligned to couple both the infrared light and the visible light from the at least one scan mirror to the eye of the user. The wearable heads-up display may include an infrared detector coupled to the support frame and aligned to detect at least a portion of infrared light returned from the eye of the user. The wearable heads-up display may include an infrared filter disposed in a position to selectively block transmission of infrared wavelengths through the holographic optical element from a side of the holographic optical element.
The wearable heads-up display may include at least one transparent lens coupled to the support frame. The holographic optical element may be embedded in a body of the at least one transparent lens. The infrared filter may be embedded in the body of the at least one transparent lens or disposed on a surface of the body of the at least one transparent lens.
The support frame may include a front frame and a pair of arms coupled to opposite sides of the front frame. The infrared detector may be mounted on one of the arms of the support frame.
The infrared detector may be aligned to receive the at least a portion of infrared light returned from the eye from the holographic optical element.
A method for tracking an eye of a user may be summarized as including outputting an infrared light, controllably reflecting the infrared light by at least one scan mirror, receiving the infrared light reflected by the at least one scan mirror at a first side of a holographic optical element and redirecting the infrared light to the eye of the user by the holographic optical element, selectively blocking transmission of infrared light through the holographic optical element from a second side of the holographic optical element by an infrared filter disposed on or proximate the second side, detecting at least a portion of infrared light returned from the eye of the user, and determining a position of at least one feature of the eye based on the at least a portion of infrared light detected from the eye of the user.
The holographic optical element in the method may include at least one hologram that is responsive to infrared light and unresponsive to visible light.
The at least a portion of infrared light returned from the eye of the user may be detected by detecting infrared light reflected from the holographic optical element.
The at least a portion of infrared light returned from the eye of the user may be detected by an infrared detector. The method may include selectively blocking transmission of infrared wavelengths to which the infrared detector is sensitive.
The infrared light outputted may be in a select subset of the infrared range, and the method may include filtering unwanted infrared wavelengths from the at least a portion of infrared light detected from the eye of the user. The unwanted infrared wavelengths may be infrared wavelengths outside of the select subset of the infrared range.
The foregoing general description and the following detailed description are exemplary of various embodiments of the invention(s) and are intended to provide an overview or framework for understanding the nature of the invention(s) as it is claimed. The accompanying drawings are included to provide further understanding of various embodiments of the invention(s) and are incorporated in and constitute part of this specification. The drawings illustrate various embodiments of the invention(s) and together with the description serve to explain the principles and operation of the invention(s).
In the drawings, identical reference numbers identify similar elements or acts. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the shapes of various elements and angles are not necessarily drawn to scale, and some of these elements are arbitrarily enlarged and positioned to improve drawing legibility. Further, the particular shapes of the elements as drawn are not necessarily intended to convey any information regarding the actual shape of the particular elements and have been solely selected for ease of recognition in the drawing.
In the following description, certain specific details are set forth in order to provide a thorough understanding of various disclosed implementations and embodiments. However, one skilled in the relevant art will recognize that implementations and embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with portable electronic devices and head-worn devices have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the implementations or embodiments. For the sake of continuity, and in the interest of conciseness, same or similar reference characters may be used for same or similar objects in multiple figures. For the sake of brevity, the term “corresponding to” may be used to describe correspondence between features of different figures. When a feature in a first figure is described as corresponding to a feature in a second figure, the feature in the first figure is deemed to have the characteristics of the feature in the second figure, and vice versa, unless stated otherwise. For the sake of continuity and conciseness, the same reference numbers may appear in multiple figures where they refer to the same features.
In this disclosure, unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open, inclusive sense, that is as “including, but not limited to.”
In this disclosure, reference to “one implementation” or “an implementation” or to “one embodiment” or “an embodiment” means that a particular feature, structures, or characteristics may be combined in any suitable manner in one or more implementations or one or more embodiments.
In this disclosure, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its broadest sense, that is, as meaning “and/or” unless the content clearly dictates otherwise.
The headings and Abstract of the disclosure provided herein are for convenience only and do not interpret the scope or meaning of the embodiments.
Throughout this specification, the term “infrared” will be understood to include “near infrared” and will generally refer to a wavelength of light that is longer than the largest wavelength of light that is typically visible to the average human eye. Light that is visible to the average human eye, i.e., “visible light” herein, is generally in the range of 400 nm-700 nm. Thus, as used herein, the term “infrared” refers to a wavelength that is greater than 700 nm, up to 1 mm.
Scanning laser projector 101 may include an optical scanner 104 that is positioned, oriented, and operable to scan the infrared light from the infrared laser diode 102 over eye 116 in a predetermined pattern. In one example, optical scanner 104 may be a scan mirror, hereafter referred to as scan mirror 104. In some examples, scanning laser projector 101 may include optics 106, 108, such as beam splitters and the like, for redirecting the infrared light provided by infrared laser diode 102 to scan mirror 104. In general, scan mirror 104 can be thought of as a mirror or reflector capable of scanning light over a surface. In one example, scan mirror 104 may be a two-dimensional scan mirror operable to scan in two directions, for example by oscillating or rotating with respect to two axes. In another example, scan mirror 104 may be two orthogonally-oriented mono-axis mirrors. The mirrors included in scan mirror 104 may be microelectromechanical systems (MEMS) mirrors, piezoelectric mirrors, and the like. In another implementation, scanning laser projector 101 may have more than one scan mirror 104. In another implementation, scanning laser projector 101 may use a mirrorless optical scanner, such as fiber optic scanner, or a combination of mirror and mirrorless optical scanning elements.
Depending on the modulation of infrared laser diode 102, scan mirror 104, or other suitable optical scanner, may scan infrared light over a substantially continuous surface of eye 116 or may scan infrared light to form an illumination pattern, such as grid pattern, crosshairs pattern, and the like, on the surface of eye 116. Generally, in order for the scan mirror 104 to scan the infrared light over eye 116, scan mirror 104 may sweep through a range of orientations. For a plurality of orientations of the scan mirror 104, scan mirror 104 may receive infrared light from infrared laser diode 102 and reflect the infrared light into a respective region of the eye 116.
System 100 may include a holographic optical element (HOE) 110 aligned to receive infrared light from scanning laser projector 101 and redirect the infrared light to eye 116. In general, HOE is an optical element that is produced using holographic principles and processes. In one implementation, HOE 110 may include a single volume of holographic material, e.g., photopolymer and/or a silver halide compound, which encodes, carries, has embedded therein or thereon, or generally includes at least one hologram 112 that is responsive to infrared light and unresponsive to visible light. “Responsive,” herein, means that the hologram redirects at least a portion of the light, where the magnitude of the portion depends on the playback efficiency of the hologram. “Unresponsive,” herein, means that the hologram transmits the light, generally without modifying the light. Hologram 112 may be responsive to the entire infrared range or to a subset of the infrared range. For example, hologram 112 may be responsive to a select infrared wavelength+/− some margin, e.g., 10% or 15% margin.
System 100 includes an infrared detector 114 positioned, oriented, and operable to detect at least a portion of a reflection of infrared light from eye 116. As used herein, the term “infrared detector” refers to a device that is sensitive and responsive to infrared light and that provides signals responsive to sensing or detecting infrared light. In one implementation, the infrared detector 114 is a photodiode or photodetector that is responsive to infrared light. In another implementation, the infrared detector 114 may be an array of photodetectors that are responsive to infrared light. In yet another implementation, the infrared detector 114 may be a complementary metal-oxide semiconductor (CMOS) camera having an array of sensors that are responsive to light in the infrared range. Although
Infrared detector 114 may detect reflections of infrared light directly and/or indirectly from eye 116. In one example, infrared detector 114 detects reflections of infrared light from HOE 110, which is aligned to receive reflections of infrared light from eye 116. In some cases, reflections of infrared light from eye 116 may reach the infrared detector 114 directly, i.e., without intervention of HOE 110. Where the infrared detector 114 is generally aligned to detect reflections of infrared light from HOE 110, the reflections that reach infrared detector 114 directly from eye 116 may be different in intensity compared to the reflections that reach infrared detector 114 from HOE 110 such that it is possible to distinguish where each detected reflection is coming from. In another example, as shown in
Returning to
HOE 110 is sufficiently optically transparent to allow ambient light 118 to pass through. This means that the infrared light detected by infrared detector 114 may be a combination of infrared light reflected from eye 116 and infrared wavelengths from the ambient light 118. The infrared light redirected to eye 116 by HOE 110 may also be influenced by infrared light from ambient light 118. This “ambient infrared light,” i.e., infrared light from sources other than the infrared laser diode 102, may induce noise in the measurements made by infrared detector 114. To reduce this noise, an infrared filter 121 is disposed in a position to block transmission of infrared light through HOE 110 from front side 111. For example, infrared filter 121 may be disposed on or proximate front side 111 to block transmission of infrared light through HOE 110 from front side 111. Infrared filter 121 may be an optically transparent coating or thin film structure or material that blocks infrared light, e.g., by reflecting and/or absorbing infrared light, while allowing transmission of other light, such as visible light. Infrared filter 121 may block all infrared wavelengths or a select subset of the infrared range. For example, if the infrared detector 114 is sensitive or responsive to a subset of the infrared range, infrared filter 121 may be selective of or block the subset of the infrared range to which the infrared detector 114 is sensitive or responsive.
In another example, an infrared filter 120 may be positioned in the receiving path between the infrared detector 114 and HOE 110, or may be integrated with infrared detector 114, to allow only select infrared wavelengths to be detected by infrared detector 114. For example, if infrared laser diode 102 works within a narrow band of the infrared range, infrared filter 120 may be selected to filter infrared wavelengths outside of this narrow band from the infrared light detected by infrared detector 114. Infrared filter 120 may be used together with infrared filter 121 in system 100 for a robust noise reduction in the measurements made by infrared detector 114.
System 100 may include one or more processors 122 that execute processor-executable instructions and one or more non-transitory storage medium or memory 124 that are communicatively coupled to the processor(s) and that store computer- or processor-executable instructions and data. Processor(s) 122 may take the form of any of a variety of analog and/or digital circuitry, for example micro-controllers, single or multi-core microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGSs), programmable logic controllers (PLUs), analog-to-digital converters (ADCs). Processor(s) 122 may include nontransitory processor readable media, for example one or more registers. Processor(s) 122 may additionally include or be coupled to other nontransitory processor readable media, for example one or more volatile memory arrays (e.g., RAM) or non-volatile memory arrays (e.g., ROM, FLASH), and/or spinning media (e.g., magnetic hard disk, optical disk).
Processor 122 may receive infrared light measurements from infrared detector 114 and use the measurements to determine a position of one or more features of eye 116 at the time the measurements were made. Processor 122 may also receive information about the orientation of the scan mirror 104 for each measurement made by the infrared detector 114 and use the orientation of the scan mirror 104 and measurement made by the infrared detector 114 to determine a position of one or more features of eye 116 at the time the measurement was made.
In one implementation, scan mirror 104 scans infrared light over eye 116 based on modulation of infrared laser diode 102, and infrared detector 114 detects an intensity pattern or map of reflected infrared light that depends on the position/orientation of eye 116. That is, each distinct orientation of scan mirror 104 may result in a respective intensity of reflected infrared light being detected by the infrared detector 114 that depends on the position/orientation of eye 116 (or the position/orientation of feature(s) of eye 116, such as the cornea, iris, pupil, retina, and so on). The intensity pattern/map detected by infrared detector 114 depends on where eye 116 is looking and enables the gaze direction and movements of eye 116 to be measured and tracked.
In one implementation, the system measures, senses, detects, identifies, or otherwise determines the intensity of infrared light reflected from eye and uses this information to identify when the intensity of a detected infrared reflection exceeds a threshold value. A detected infrared reflection that exceeds the threshold value is used herein because such generally corresponds to a spectral reflection for the eye known as the first Purkinje image or glint. The glint provides a useful, reliable, and sufficient detection feature for the purpose of determining the gaze direction of the eye of the user. Thus, only detected reflections that correspond to glints are used to determine the gaze direction of the eye. However, the entire collection of detected reflections of the infrared light from the eye can be useful in other applications.
System 200 includes HOE 210 having at least one infrared hologram 212 that is responsive to infrared light and unresponsive to visible light and at least one visible hologram 232 that is responsive to visible light and unresponsive to infrared light, where responsive and unresponsive has the meaning previously described with respect to hologram 112 (in
When HOE 210 redirects visible light, hologram 232 may apply a first optical power to the visible light. The first optical power may be a positive optical power that focuses or converges the visible light to, for example, an exit pupil having a diameter less than 1 cm at eye 216 for the purpose of providing a clear and focused image with a wide field of view. When HOE 210 redirects infrared light, the hologram 212 may apply a second optical power to the infrared light. The second optical power may be less than the first optical power so that the hologram 212 redirects infrared light over an area of eye 216 that is larger than the exit pupil of visible light.
System 200 includes an infrared detector 214 positioned, oriented, and operable to detect infrared light reflected from HOE 210. In the example shown in
Returning to
In operation, scan mirror 204 repeatedly scans, oscillates, pivots or rotates over a range of orientations (e.g., entire range of orientations) while the visible laser diodes R, G, B provide visible light corresponding to pixels of a scanned image or, generally, “displayed content.” At the same time, infrared laser diode 202 may be activated to illuminate the eye with infrared light. Depending on the implementation, infrared laser diode 202 may be modulated to provide an illumination pattern, e.g., grid, a set of parallel lines, a crosshair, or any other shape/pattern, on eye 216. Because infrared light is invisible to eye of the user, infrared light does not interfere with the scanned image being projected with the visible light. The infrared light scanned over the eye 216 is reflected off the eye 216. At least a portion of the infrared light reflected off the eye 216 is received by HOE 210 and reflected to infrared detector 214. Infrared detector 214 detects an intensity pattern or map of the reflected infrared light that depends on the position/orientation of eye 216. For each orientation of the scan mirror 204, processor 222 uses the measurement by infrared detector 214 to determine a position of one or more features of eye 216.
Scanning laser projector 201 is mounted on arm 344. Scan mirror 204 of scanning laser projector 201 is aligned with HOE 210 such that HOE 210 can receive infrared light and visible light from scan mirror 204. In the illustrated example, HOE 210 receives infrared light and visible light directly from scan mirror 204. In other examples, alignment between scan mirror 204 and HOE 210 may include light directing optics. Such light directing optics may be needed, for example, if due to space constraints it is not possible to place the scan mirror 204 where a direct line of sight between the scan mirror 204 and HOE 210 that avoids obscuring features such as eyelids and eyelashes can be established. In other examples, beam shaping optics or beam replicating optics may be arranged in a path through which HOE 210 receives infrared light and visible light from scan mirror 204. For example, it may be desired to use a focusing lens to focus the scanning light from scan mirror 204 onto HOE 210.
Infrared detector 214 is mounted on arm 344. In the example shown in
Processor(s) 222 and storage medium/media (or memory) 224 may be mounted on either of the arms 344, 346, subject to mounting space availability. In the example of
In use, laser module 203 provides both visible light and infrared light. Scan mirror 204 moves (e.g., oscillates) to scan both visible light and infrared light over eye 316 by scanning both visible light and infrared light over HOE 210, as explained for system 200 (in
In the implementation where infrared filter 421 is disposed on front side 411 of HOE 410, infrared filter 421 may be in the form of a coating or thin film disposed on front side 411 of HOE 410. The coating or thin film may be monolayer or multilayer. For example, infrared filter 421 may be in the form of a multilayer dielectric coating or thin film that selectively reflects infrared light while allowing the remainder of ambient light, such as visible light, to pass through. The method of disposing infrared filter 421 on front side 411 of HOE 410 may include, for example, deposition, coating, and/or lamination processes. In general, the method of disposing may include any method of keeping the infrared filter 421 and HOE 410 as an integral unit.
In the illustrated example, optical filter 604 is embedded in lens body 602. The embedding process may involve molding lens body 602 around optical filter 604. In one example, embedded means that the optical filter 604 is wholly contained within the lens body 602, as shown in
Lens 600 appears flat in
Optical filters 400, 500, as described in
The above description of illustrated embodiments, including what is described in the Abstract of the disclosure, is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Although specific embodiments and examples are described herein for illustrative purposes, various equivalent modifications can be made without departing from the spirit and scope of the disclosure, as will be recognized by those skilled in the relevant art. The teachings provided herein of the various embodiments can be applied to other portable and/or wearable electronic devices, not necessarily the exemplary wearable electronic devices generally described above.
This application claims benefit of U.S. Provisional Application No. 62/632,785, filed Feb. 20, 2018, the disclosure of which is incorporated herein in its entirety by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20160018639 | Spitzer | Jan 2016 | A1 |
| 20160349514 | Alexander | Dec 2016 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20190258062 A1 | Aug 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62632785 | Feb 2018 | US |
