Patent Application
20230105302
This application claims priority to Swedish Application No. 2151198-5, entitled “GAZE DEFECT COMPENSATION,” filed on Sep. 30, 2021. The entire disclosure of the above-referenced application is incorporated herein by this reference.
The present disclosure generally relates to the field of eye tracking. In particular, the present disclosure relates to eye tracking systems and methods for providing accurate eye tracking in the presence of health-related eye defects.
In eye tracking applications, digital images are retrieved of the eyes of a user and the digital images are analyzed in order to estimate the gaze direction of the user. The estimation of the gaze direction may be based on computer-based image analysis of features of the imaged eye. One known example method of eye tracking includes the use of infrared light and an image sensor. The infrared light is directed towards the pupil of a user and the reflection of the light is captured by an image sensor.
Many eye tracking systems estimate gaze direction based on identification of a pupil position together with glints or corneal reflections. Eye tracking systems may include a calibration sequence for defining an eye tracking model that can map the pupil position and glints to gaze direction. However, such eye tracking models can suffer from poor performance for users with health-related eye defects.
Portable or wearable eye tracking devices have also been previously described. One such eye tracking system is described in U.S. Pat. No. 9,041,787 (which is hereby incorporated by reference in its entirety). A wearable eye tracking device is described using illuminators and image sensors for determining gaze direction.
According to a first aspect of the present disclosure there is provided an eye tracking system comprising a controller configured to, for one or both of a left eye and a right eye of a user:
The eye tracking system can advantageously implement a gaze dependent eye-model that applies a different calculation process depending on a direction of the optical axis / user’s gaze. By selecting one of a plurality of eye models based on a direction of the optical axis, the eye tracking system can advantageously compensate for Strabismus or other gaze related defects for one or both eyes.
The controller may be configured to select one of a plurality of predetermined eye models based on the direction of the optical axis.
The controller may be configured to select one of a plurality of eye models by: selecting a gaze offset value from a plurality of gaze offset values based on the direction of the optical axis; and applying the selected gaze offset value to a baseline eye model.
The controller may be configured to select one of a plurality of eye models by: selecting a gaze offset value from a plurality of gaze offset values based on the direction of the optical axis; and applying the selected gaze offset value to the selected predetermined eye model.
The controller may be configured to determine the plurality of eye models by determining, during a calibration process: a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models; a baseline eye model and a plurality of gaze offset values for applying to the baseline eye model; or a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models and a plurality of gaze offset values for applying to one of the predetermine eye models.
The controller may be configured to determine the plurality of eye models by determining, during a calibration process: a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models for a corresponding plurality of stimulus points or regions of stimulus points; a baseline eye model and a plurality of gaze offset values for a corresponding plurality of stimulus points or regions of stimulus points, wherein the gaze offset values are for applying to the baseline eye model; or a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models for a corresponding plurality of regions of stimulus points, and a plurality of gaze offset values for corresponding subsets of the regions of stimulus points, wherein the gaze offset values are for applying to one of the predetermined eye models.
The controller may be configured to: cause a plurality of stimulus points to be displayed one at a time to a user; and receive eye measurement data for each stimulus point.
The plurality of stimulus points may comprise 6 or more stimulus points.
Each of: the plurality of eye modelling parameter sets; and / or the plurality of gaze offset values, may correspond to: each stimulus point; or a region of stimulus points.
The controller may be configured to: determine a gaze vector for the left eye of the user; determine a gaze vector for the right eye of the user; determine a weighting for each of the left eye gaze vector and the right eye gaze vector based on the selected eye model for the respective left and right eye; and apply the weighting to the gaze vector of each of the left eye and the right eye to provide a combined gaze vector.
The controller may be configured to determine the weighting based on a magnitude of the gaze offset value associated with the selected eye model.
The controller may be configured to determine the weighting as dependent on a variation of the gaze offset values associated with the selected eye model relative to neighboring values of the plurality of gaze offset values.
The controller may be configured to determine the weighting as dependent on a variation of the values of the eye modelling parameter sets associated with the selected eye model relative to neighboring values of the plurality of eye modelling parameter sets.
The controller may be configured to determine a plurality of weightings during the calibration process, each weighting corresponding to each of: the plurality of eye modelling parameter sets; the plurality of gaze offset values; and / or the plurality of stimulus points.
Each weighting may comprise a value from 0 to 1.
The controller may be further configured to: cause the plurality of stimulus points to be re-displayed one at a time to the user; and for each stimulus point that is displayed: receive eye measurement data; select the eye model corresponding to the stimulus point; calculate a gaze vector using the selected eye model and the eye measurement data; calculate a difference between the calculated gaze vector and the known gaze vector corresponding to the stimulus point; and determine a weighting based on the difference.
According to a second aspect of the present disclosure there is provided a head-mounted device comprising any eye tracking system as disclosed herein.
According to a third aspect of the present disclosure, there is provided a method of eye tracking, the method comprising:
According to a fourth aspect of the present disclosure, there is provided a method of calibrating an eye tracking system, the method comprising:
According to a fifth aspect of the present disclosure there is provided one or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed by a computing system, causes the computing system to perform any method disclosed herein.
There may be provided a computer program, which when run on a computer, causes the computer to configure any apparatus, including a circuit, controller or device disclosed herein or perform any method disclosed herein. The computer program may be a software implementation, and the computer may be considered as any appropriate hardware, including a digital signal processor, a microcontroller, and an implementation in read only memory (ROM), erasable programmable read only memory (EPROM) or electronically erasable programmable read only memory (EEPROM), as non-limiting examples. The software may be an assembly program.
The computer program may be provided on a computer readable medium, which may be a physical computer readable medium such as a disc or a memory device, or may be embodied as a transient signal. Such a transient signal may be a network download, including an internet download. There may be provided one or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed by a computing system, causes the computing system to perform any method disclosed herein.
One or more embodiments will now be described by way of example only with reference to the accompanying drawings in which:
The eye tracking system 100 may comprise circuitry or one or more controllers 125, for example including a receiver 126 and processing circuitry 127, for receiving and processing the images captured by the image sensor 120. The circuitry 125 may for example be connected to the image sensor 120 and the optional one or more illuminators 110-119 via a wired or a wireless connection and be co-located with the image sensor 120 and the one or more illuminators 110-119 or located at a distance, e.g. in a different device. In another example, the circuitry 125 may be provided in one or more stacked layers below the light sensitive surface of the light sensor 120.
The eye tracking system 100 may include a display (not shown) for presenting information and/or visual stimuli to the user. The display may comprise a VR display which presents imagery and substantially blocks the user’s view of the real-world or an AR display which presents imagery that is to be perceived as overlaid over the user’s view of the real-world.
The location of the image sensor 120 for one eye in such a system 100 is generally away from the line of sight for the user in order not to obscure the display for that eye. This configuration may be, for example, enabled by means of so-called hot mirrors which reflect a portion of the light and allows the rest of the light to pass, e.g. infrared light is reflected, and visible light is allowed to pass.
While in the above example the images of the user’s eye are captured by a head-mounted image sensor 120, in other examples the images may be captured by an image sensor that is not head-mounted. Such a non-head-mounted system may be referred to as a remote system.
In an eye tracking system, a gaze signal can be computed for each eye of the user (left and right). The quality of these gaze signals can be reduced by disturbances in the input images (such as image noise) and by incorrect algorithm behavior (such as incorrect predictions). A goal of the eye tracking system is to deliver a gaze signal that is as good as possible, both in terms of accuracy (bias error) and precision (variance error). For many applications it can be sufficient to deliver only one gaze signal per time instance, rather than both the gaze of the left and right eyes individually. Further, the combined gaze signal can be provided in combination with the left and right signals. Such a gaze signal can be referred to as a combined gaze signal.
The system may employ image processing (such as digital image processing) for extracting features in the image. The system may for example identify a position of the pupil 230 in the one or more images captured by the image sensor. The system may determine the position of the pupil 230 using a pupil detection process. The system may also identify corneal reflections 232 located in close proximity to the pupil 230. The system may estimate a corneal center or eyeball center based on the corneal reflections 232. For example, the system may match each of the individual corneal reflections 232 for each eye with a corresponding illuminator and determine the corneal center of each eye based on the matching.
The eye tracking system may perform a calibration routine to estimate the PPO and FO for each individual user, prior to eye tracking. During calibration, one or more stimulus points (for example 5) at known positions / gaze angles may be displayed in turn on a screen and the user may be instructed to direct their gaze to each point as it is displayed. The eye tracking system can then compare the determined optical axis 336 to the true known gaze direction 338 (assuming that the user is looking at the correct stimulus point) to determine the PPO 340 and the FO 342. In this way, the eye tracking system can determine a personalized eye gaze tracking model (referred to herein as eye model) for each user based on the determined constants PPO 340 and FO 342.
The system can then determine a gaze vector (which may also be referred to as a gaze ray) for each eye. In some examples, the gaze vector may be based on a gaze origin and gaze direction which can be determined from the respective glint to illuminator matching / corneal centers, the determined pupil position and the calibrated constants (as defined by the eye tracking model). The gaze vector for each eye may be combined to provide a combined gaze vector.
The above-described process for determining the gaze vector is an example of a pupil center corneal reflection (PCCR) approach. It will be appreciated that variations and alternatives to the above approach for determining an eye model for images of a user’s eye are known in the art and that the present disclosure is not limited to any specific approach.
The above approach for determining a single calibrated eye model for each user can suffer performance issues when a user has a health-related eye defect. One such eye defect is Strabismus or “Lazy-Eye.” 4% of the population suffers from Strabismus, which is characterized by an inability to always rotate both eyes towards a point of focus. The effect can be varied across individuals, which creates an additional layer of difficulty for eye-tracking. The effect can appear as: improper movement of one or both eyes in one or more directions; limited range of movement of one or both eyes; scaled movement relative to point of focus for one or both eyes; a constant offset in gaze for one eye; different constant offsets in both eyes depending on if the user is looking to the left or right; both eyes being cross-eyed (converged too close), non-constant offsets and / or many other effects. In some examples, Strabismus is non-uniform with effects varying greatly across the range of motion of the eye. In some examples Strabismus is inconsistent, in that the same stimulus point may not necessarily result in the same eye movement.
As a result of the varied nature of Strabismus, the personalized calibrated eye models described above suffer from performance issues for users exhibiting Strabismus. One approach for addressing Strabismus is for a user to disable the tracking for an affected eye. However, this approach severely limits the performance for unaffected gaze directions as combined gaze performance is better than single eye performance in most cases. This approach also fails to compensate for user’s with Strabismus in both eyes.
The systems and methods disclosed below can provide eye tracking that overcomes the above limitations and can compensate for Strabismus or other personal gaze defects. The disclosed systems and methods can advantageously implement a gaze dependent eye-model that applies a different calculation process depending on a direction of the optical axis / user’s gaze.
The eye measurement data may include one or more parameters determined from an image of the eye of the user. For example, the eye measurement data may comprise the position of corneal reflections, a calculated corneal center, a determined pupil position etc., as described above. In other examples, the eye measurement data may comprise a raw image of the eye received from an image sensor. In such examples, the controller 552 is further configured to determine the corneal center and pupil position from the raw image as described above and known in the art.
By selecting one of a plurality of eye models based on a direction of the optical axis, the eye tracking system can advantageously compensate for Strabismus for one or both eyes. In particular, the eye tracking system may implement different eye models (and therefore perform different gaze calculation processes) for Strabismus regions and regions of normal eye behavior.
The approaches described above use a few (for example, 5) known stimulus points to calibrate a single eye-model with eye model parameters (e.g. PPO, FO). As a result, the same eye model is used for all gaze angles. The eye tracking system 550 of
The plurality of eye models may be determined from a calibration routine, such as that illustrated in
In a first example, the plurality of eye models may comprise a plurality of predetermined eye models determined by the controller 552 as part of the calibration process. Such eye models can be considered as predetermined in that they are determined during a calibration process, prior to use of the eye tracking system on “live” (i.e. non-calibration) eye measurement data. As an example, during calibration, the controller 552 may determine individual eye models for each stimulus point 660 based on eye measurement data that was recorded when the user was instructed to look at each corresponding stimulus point 660. As a further example, the controller 552 may determine individual eye models for a plurality of regions of stimulus points 660. For example, the controller 552 may determine four individual eye models, one corresponding to each quadrant 664a, 664b, 664c, 664d (left upper, right upper, left lower, right lower, referred to as quadrants 664) as viewed by a user, as illustrated in
During use of the eye tracking system 550 (following calibration), the optical axis detector 553 can determine a direction of the optical axis of the eye in the usual manner, for example based on the PCCR method, to determine an approximate gaze angle of the user. In this way, the controller 552 can determine which stimulus point 660 or region of stimulus points (e.g. which quadrant 664a-664d) is associated with the determined optical axis. The gaze estimation model 554 can then select the eye model corresponding to the determined region / quadrant 664a-664d / stimulus point 660. In this way, the controller 552 can use an eye-model with different eye model parameters (PPO, FO etc.) based on the determined angle of the optical axis, to account for Strabismus effects or similar eye gaze defects. If a direction of the optical axis is between two or more stimulus points 660, the controller 552 may interpolate the model parameters for the eye-models corresponding to the closest stimulus points 660. The interpolation may depend on a distance to each stimulus point 660 such that the closer stimulus point 660 is more heavily weighted in the combination.
In a second example, the plurality of eye models may comprise a baseline eye model and a plurality of gaze offset values. The controller 552 may determine the baseline eye model and the plurality of gaze offset values as part of the calibration process. The controller 552 may determine the baseline eye model from one or more stimulus points 660, for example from 5 stimulus points 660 or all stimulus points 660, using the conventional approach. Each of the plurality of gaze offset values may correspond to a respective stimulus point 660 or region of stimulus points 660. The controller 552 may determine each gaze offset value as an error value between a calculated baseline gaze angle using the baseline eye model and an expected gaze angle for the respective stimulus point or points 660. Each gaze offset value may comprise an error value in the vertical direction (y-axis) and an error value in the horizontal direction (x-axis). The plurality of gaze offset values may be referred to as an error value matrix (EVM). The size of the EVM may correspond to the size of the stimulus point grid or the number of distinct regions of stimulus points. In this way, the controller is configured to determine a plurality of gaze offset values for applying to a baseline eye model during the calibration process.
During use, following calibration, the optical axis detector 553 can determine a direction of the optical axis. The gaze estimation module 552 may then determine the stimulus point 660 (or region of stimulus points) that is closest to the determined direction of the optical axis. The gaze estimation module 552 can then select one of a plurality of eye models by selecting the gaze offset value corresponding to the stimulus point 660 closest to the direction of the optical axis, and applying the gaze offset value to the baseline model. In this way, the controller 552 can use a single eye-model with the same eye model parameters (PPO, FO etc.) and apply a direction dependent gaze angle offset to account for Strabismus or similar defects. If a direction of the optical axis is between two or more stimulus points 660 (or regions of stimulus points), the controller 552 may interpolate the error vector applied to the calculated baseline gaze angle based on the gaze offset values from the closest stimulus points 660. The interpolation may depend on a distance to each stimulus point 660 or region of stimulus points 660.
The second example maintains a single value for each of the PPO and FO in the baseline model, which provides a single representation of the real physical parameters of the eye. This can advantageously avoid anomalously large numbers of PPO or FO, which can result in undesirable noise sensitivity. This approach can also advantageously provide a simpler approach than the first example by providing a single eye model with a plurality of scalar offsets.
A third example may comprise a blend of the first and second examples. In other words, the controller 552 may determine a plurality of predetermined eye models and a plurality of gaze offset values during the calibration process. Each predetermined eye model may correspond to a different region of calibration / stimulus points, for example there may be a separate eye model for each of the four quadrants 664. The controller 552 may determine each gaze offset value for each individual stimulus point in the region as an error value between a calculated gaze angle using the appropriate predetermined eye model and a known gaze angle of the respective stimulus point 660. For instance, the controller may determine an individual gaze offset value for each stimulus point in the region / quadrant, relative to the predetermined eye model for that region / quadrant. In this way, the controller is configured to determine a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models; and a plurality of gaze offset values for applying to one of the predetermined eye models, during the calibration process.
During subsequent use, the gaze estimation module 554 may select one of a plurality of eye models by: selecting one of a plurality of predetermined eye models based on a direction of the optical axis; selecting one of a plurality of gaze offset values based on a direction of the optical axis; and applying the selected gaze offset value to the selected predetermined eye model.
In some instances of Strabismus, eye behavior may be inconsistent for one or more gaze angles. In some examples, the eye tracking system 550 may be configured to detect such inconsistent eye-performance such that eye tracking may be disabled for affected gaze angles for the affected eye.
It will be appreciated that the controller can perform the above calibration process, model selection and gaze vector determination for both the left eye and the right eye of the user. The controller 552 can combine the resultant eye vectors for each eye to provide a combined gaze vector. In some examples, the controller 552 may apply weightings to the left eye gaze vector and the right eye gaze vector when providing the combined gaze vector. The controller can determine the weighting for each eye based on values associated with the selected eye model for the respective eye. For example, for the second and third examples described above, the weightings may be based on a magnitude of the gaze offset value, which represents an error of the user’s gaze relative to the baseline or predetermined model. The weightings may also be based on a variation of the gaze offset value relative to neighboring gaze offset values (associated with neighboring stimulus points). High variation in gaze offset values for a particular region (of stimulus points 660) may indicate that the eye gaze vector is erratic in that region and therefore may be an indication of Strabismus or other gaze related defects. For the first and third examples, the weightings may be based on a variation of values of the eye modelling parameter sets (e.g. PPO, FO) relative to neighboring values of the plurality of eye modelling parameter sets (associated with neighboring stimulus points / regions of stimulus points). The weightings may comprise a value from 0 to 1. The sum of the pair of weightings (one for each eye) may equal 1. The weightings may depend on the magnitude of the gaze offset value or the size of any variation. For regions of particularly high magnitude gaze offset values (magnitude above a threshold) or particularly high variation (variation above a threshold variation) of the gaze offset values and / or values of eye modelling parameter sets, the controller 552 may set the corresponding weightings to zero to disable eye gaze tracking in that region. In some examples, both eyes may experience the same regions of poor performance and the controller may set the corresponding weightings for both eyes to zero, disabling eye tracking altogether. In this way, the eye tracking system 550 can advantageously disable eye tracking rather than provide an inaccurate gaze value in regions of severe Strabismus. In other examples, the controller 552 may only set the weighting to zero for the eye with the highest error / variation.
In some examples, the controller 552 may determine a plurality of weightings (which may also be referred to as pairs of weighting values - one weighting value for each eye) during the calibration process. Each of the plurality of weightings may correspond to: each of the plurality of predetermined models / eye modelling parameter sets (examples 1 and 3); each of the plurality of gaze offset values (examples 2 and 3); and / or each stimulus point 660 (examples 1, 2 and 3).
The plurality of weightings may be referred to as an eye weighting matrix (EWM) with a size corresponding to the number of different eye models (which can be implemented as a plurality of different eye models and / or a plurality of different gaze offset values), which can be same as the size of the stimulus point grid.
In some examples, the controller 552 may determine the plurality of weighting values as part of a two-stage calibration process. During a first stage, a plurality of stimulus points 660 may be displayed one at a time to the user 662 and the controller 552 may determine (for each eye) a plurality of eye modelling parameter sets; and / or a plurality of gaze offset values, corresponding to each stimulus point or region of stimulus points as described above in relation to examples 1 to 3. During the second stage, the plurality of stimulus points 660 may be re-displayed one at a time to the user 662. As each stimulus point is displayed, the controller 552 may, for each eye, receive eye measurement data, select the eye model corresponding to the stimulus point and determine a calculated gaze vector using the selected eye model and eye measurement data. The controller 552 may then compare the calculated eye vector for each eye to the known eye vector corresponding to the stimulus point and determine a weighting for each eye based on a magnitude of the difference between the calculated eye vector and the known eye vector. In this way, selected eye models that do not accurately calculate the gaze vector can be given a lower weighting. This can be useful when subsequently combining gaze vectors for the left and right eyes such that gaze vectors that are expected to be less accurate have a lesser influence on the combined gaze signal. As described above, one or more weighting values for one or both eyes may be set to zero to disable gaze tracking for the relevant eye(s) during subsequent use when the user’s gaze corresponds to the direction of the relevant stimulus point(s). A weighting value may be set to zero if the difference between the calculated eye vector and the known eye vector is greater than an acceptable difference threshold.
In a similar example, the first stage of the calibration process may comprise the controller 552 determining (for each eye) a first plurality of gaze offset values for a plurality of stimulus points 660 for applying to a baseline model (example 2) or one of a plurality of predetermined models (example 3). The second stage may comprise the controller 552 determining (for each eye) a second plurality of gaze offset values for the same plurality of stimulus points for applying to the same baseline model or the same one of a plurality of predetermined models. The difference between each of the first plurality of gaze offset values and the corresponding one of the second plurality of gaze offset values represents the consistency of the user’s eye gaze towards that stimulus point / region of stimulus points. The controller 552 can then determine (for each eye) a plurality of weightings based on the difference between corresponding values of the gaze offset values in the first and second plurality of gaze offset values.
Step 770 comprises receiving eye measurement data associated with an eye of a user. Step 772 comprises determining an optical axis of the eye from the eye measurement data. Step 774 comprises selecting one of a plurality of eye models based on a direction of the optical axis. Step 776 comprises determining a gaze vector of the eye by applying the selected eye model to the eye measurement data.
Step 880 comprises causing a plurality of stimulus points to be displayed one at a time to a user. Step 882 comprises receiving eye measurement data for each stimulus point. Step 884 comprises determining: a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models; a baseline eye model and a plurality of gaze offset values for applying to the baseline eye model; or a plurality of eye modelling parameter sets defining a corresponding plurality of predetermined eye models and a plurality of gaze offset values for applying to one of the predetermine eye models. Each of: the plurality of eye modelling parameter sets; and / or the plurality of gaze offset values correspond to: each stimulus point; or a region of stimulus points.
The disclosed eye tracking systems can provide a strabismus compensation mode by implementing the above-described calibration process, model definition and model selection. A user suffering from gaze defects may select the strabismus compensation mode instead of the conventional single model personal calibration.
The disclosed system and methods provide a solution for users to provide functioning eye-tracking that accounts for their individual strabismus effect.
Throughout the present specification, the descriptors relating to relative orientation and position, such as “horizontal”, “vertical”, “top”, “bottom” and “side”, are used in the sense of the orientation of the apparatus / device as presented in the drawings. However, such descriptors are not intended to be in any way limiting to an intended use of the described or claimed invention.
It will be appreciated that any reference to “close to”, “before”, “shortly before”, “after” “shortly after”, “higher than”, or “lower than”, etc., can refer to the parameter in question being less than or greater than a threshold value, or between two threshold values, depending upon the context.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2151198-5 | Sep 2021 | SE | national |
