Patent Application
20230290290
The present invention generally relates to color correction for waveguides and, more specifically, to real-time color correction for waveguide applications.
Waveguides can be referred to as structures with the capability of confining and guiding waves (e.g., restricting the spatial region in which waves can propagate). One subclass includes optical waveguides, which are structures that can guide electromagnetic waves, typically those in the visible spectrum. Waveguide structures can be designed to control the propagation path of waves using a number of different mechanisms. For example, planar waveguides can be designed to utilize diffraction gratings to diffract and couple incident light into the waveguide structure such that the in-coupled light can proceed to travel within the planar structure via total internal reflection (TIR).
Fabrication of waveguides can include the use of material systems that allow for the recording of holographic optical elements within the waveguides. One class of such material includes polymer dispersed liquid crystal (PDLC) mixtures, which are mixtures containing photopolymerizable monomers and liquid crystals. A further subclass of such mixtures includes holographic polymer dispersed liquid crystal (HPDLC) mixtures. Holographic optical elements, such as volume phase gratings, can be recorded in such a liquid mixture by illuminating the material with two mutually coherent laser beams. During the recording process, the monomers polymerize, and the mixture undergoes a photopolymerization-induced phase separation, creating regions densely populated by liquid crystal micro-droplets, interspersed with regions of clear polymer. The alternating liquid crystal-rich and liquid crystal-depleted regions form the fringe planes of the grating. The resulting grating, which is commonly referred to as a switchable Bragg grating (SBG), has all the properties normally associated with volume or Bragg gratings but with much higher refractive index modulation ranges combined with the ability to electrically tune the grating over a continuous range of diffraction efficiency (the proportion of incident light diffracted into a desired direction). The latter can extend from non-diffracting (cleared) to diffracting with close to 100% efficiency.
Waveguide optics, such as those described above, can be considered for a range of display and sensor applications. In many applications, waveguides containing one or more grating layers encoding multiple optical functions can be realized using various waveguide architectures and material systems, enabling new innovations in near-eye displays for augmented reality (AR) and virtual reality (VR), compact head-up displays (HUDs) and helmet-mounted displays or head-mounted displays (HMDs) for road transport, aviation, and military applications, and sensors for biometric and laser radar (LIDAR) applications.
Various embodiments are directed to a color correction system including: an eye tracker system; and an optical engine controller in communication with the eye tracker system and an optical engine, where the eye tracker system is configured to compute eye gaze direction vectors and eye pupil coordinates, and where the optical engine controller is configured to compute a compensation factor based on the eye gaze direction vectors and the eye pupil coordinates.
In various other embodiments, the eye tracker system includes: a light source; a detector; a waveguide comprising: an input grating to inputting illumination light from the light source into the waveguide; an illumination grating for deflecting the illumination light towards an eye; an detector grating to input illumination light reflected off the eye into the waveguide; and an output grating for outputting the reflected illumination light from the waveguide into the detector, wherein the detector is configured to compute the eye gaze direction vectors and the eye pupil coordinates based on reflected illumination light.
In still various other embodiments, the illumination grating and/or the imaging grating includes an array of switchable beam deflection grating elements.
In still various other embodiments, the array of switchable beam deflection grating elements includes grating elements elongated with a longer dimension orthogonal to a beam propagation direction of the illumination light.
In still various other embodiments, the array of switchable beam deflection grating elements includes grating elements disposed in a two-dimensional array.
In still various other embodiments, each grating element is configured to have optical power in two orthogonal planes.
In still various other embodiments, the illumination grating and the imaging grating comprises a single grating configured to both deflect the illumination light towards the eye and input illumination light reflected off the eye into the waveguide.
In still various other embodiments, the illumination grating and the imaging grating include separate gratings in separate layers.
In still various other embodiments, the illumination grating and the input grating are in a first grating layer and the imaging grating and the output grating are in a second grating layer.
In still various other embodiments, the first grating layer and second grating layer are separate grating layers and are each sandwiched by substrate forming two distinct waveguiding structures.
In still various other embodiments, the illumination grating and the imaging grating respond to different total internal reflection angle ranges within the waveguide.
In still various other embodiments, the illumination light is reflected off the front surface of the cornea, the lens, and/or the retina of the eye.
In still various other embodiments, the illumination light includes infrared light.
In still various other embodiments, the optical engine controller uses a look-up table to compute the compensation factor.
In still various other embodiments, the look-up table correlates different gaze directions and pupil coordinates of the user's eye with different image compensation factors.
In still various other embodiments, the compensation factor includes color correction or image correction.
In still various other embodiments, the detector includes a detector signal processor which is configured to compute the eye gaze direction vectors and the eye pupil coordinates from the detected illumination light.
Further, various embodiments are directed to a waveguide-based display including: the color correction system as disclosed above; and another waveguide configured to accept light from the optical engine and output the light into the user's eye.
In various other embodiments, the waveguide and the other waveguide are in contact with each other.
In still various other embodiments, the waveguide and the other waveguide are separated by an air gap or a low reflective material.
Further, various embodiments are directed to a method of providing real-time color correction for a waveguide based display, the method including: transmitting a light pattern onto a user's eye; reflecting the light pattern off of a feature of the user's eye; directing the reflected light pattern towards a detector to produce eye tracking information, wherein the eye tracking information comprises eye gaze direction vectors and eye pupil coordinates; determining the user's viewing condition based on the eye tracking information; based on the determined viewing condition, applying a compensation factor to an input image light.
In various other embodiments, the feature of the user's eye includes the lens, the cornea, or the retina of the user's eye.
In still various other embodiments, compensation factor includes compensation to an input image to a waveguide.
In still various other embodiments, the compensation factor is used to alter an input image originating from an optical engine.
In still various other embodiments, the compensation factor is applied with frequency that is equal to or less than the operating frequency of the detector.
In still various other embodiments, the compensation factor includes color correction or image correction.
The description will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.
For the purposes of describing embodiments, some well-known features of optical technology known to those skilled in the art of optical design and visual displays have been omitted or simplified in order to not obscure the basic principles of the invention. Unless otherwise stated, the term “on-axis” in relation to a ray or a beam direction refers to propagation parallel to an axis normal to the surfaces of the optical components described in relation to the invention. In the following description the terms light, ray, beam, and direction may be used interchangeably and in association with each other to indicate the direction of propagation of electromagnetic radiation along rectilinear trajectories. The term light and illumination may be used in relation to the visible and infrared bands of the electromagnetic spectrum. Parts of the following description will be presented using terminology commonly employed by those skilled in the art of optical design. As used herein, the term grating may encompass a grating comprised of a set of gratings in some embodiments. For illustrative purposes, it is to be understood that the drawings and figures are not drawn to scale unless stated otherwise.
Optical waveguides utilizing nanoscale optical features can be implemented to provide transparent displays for various applications. In many display applications, it would be ideal that every pixel rendered with the same color (RGB) value would look identical in the output of the waveguide. In practice, brightness, contrast, color, and other properties can vary in the output due to certain limitations that can depend on the grating design. For example, one common issue in the field of waveguide displays is color non-uniformity in the final output image. Furthermore, such problems can be compounded as these properties can also vary depending on the user's viewing conditions. In many cases, these variations can be minimized or corrected for during processing and/or by optimizing the waveguide design. Correction during processing can be implemented by pre-calibrating the waveguide by measuring the output image and then altering the input image light to compensate for the variations. However, addressing non-uniformity issues that vary based on the user's viewing conditions during processing can be difficult or impossible since correcting for one viewing condition may affect another. As such, many embodiments of the invention are directed towards resolving or mitigating the issue of color non-uniformity in waveguides in real-time based on the user's viewing condition. Real-time color correction systems and methods in accordance with various embodiments of the invention can be utilized as appropriate with a variety of different waveguides for various applications, including but not limited to waveguide displays. Real-time color correction systems and methods may also be used to mitigate other types of photometric nonuniformity in real time based on the user's viewing condition. For example, luminance non-uniformity and contrast non-uniformity may be mitigated.
In many embodiments, the correction system utilizes one or more eye trackers to determine the user's viewing condition to provide an appropriate real-time correction or compensation to the output image. The information can be conveyed to a controller, which can send signals to the projection system to project the corrected image. As can readily be appreciated, various types of eye trackers along with algorithms for implementing such tools can be utilized as appropriate depending on the specific requirements of a given application. In some embodiments, the eye tracker utilizes a projector for projecting a known pattern of light onto the eye. Reflections and changes in the reflections can be used to infer eye rotation, pupil location, and other ocular information. In some embodiments, features of the eyes are tracked over time to infer eye rotation through changes in the reflections. In many embodiments, cameras and optical sensors are utilized to sense light, typically infrared, that is reflected from the eye to implement eye tracking. Waveguides, waveguide displays, optical structures and configurations, eye tracking solutions, and systems and methods for real-time color correction are described below in further detail.
Optical structures recorded in waveguides can include many different types of optical elements, such as but not limited to diffraction gratings. Gratings can be implemented to perform various optical functions, including but not limited to coupling light, directing light, and preventing the transmission of light. In many embodiments, the gratings are surface relief gratings that reside on the outer surface of the waveguide. In other embodiments, the grating implemented is a Bragg grating (also referred to as a volume grating), which are structures having a periodic refractive index modulation. Bragg gratings can be fabricated using a variety of different methods. One process includes interferential exposure of holographic photopolymer materials to form periodic structures. Bragg gratings can have high efficiency with little light being diffracted into higher orders. The relative amount of light in the diffracted and zero order can be varied by controlling the refractive index modulation of the grating, a property that can be used to make lossy waveguide gratings for extracting light over a large pupil.
One class of Bragg gratings used in holographic waveguide devices is the Switchable Bragg Grating (SBG). SBGs can be fabricated by first placing a thin film of a mixture of photopolymerizable monomers and liquid crystal material between substrates. The substrates can be made of various types of materials, such glass and plastics. In many cases, the substrates are in a parallel configuration. In other embodiments, the substrates form a wedge shape. One or both substrates can support electrodes, typically transparent tin oxide films, for applying an electric field across the film. The grating structure in an SBG can be recorded in the liquid material (often referred to as the syrup) through photopolymerization-induced phase separation using interferential exposure with a spatially periodic intensity modulation. Factors such as but not limited to control of the irradiation intensity, component volume fractions of the materials in the mixture, and exposure temperature can determine the resulting grating morphology and performance. As can readily be appreciated, a wide variety of materials and mixtures can be used depending on the specific requirements of a given application. In many embodiments, HPDLC material is used. During the recording process, the monomers polymerize, and the mixture undergoes a phase separation. The LC molecules aggregate to form discrete or coalesced droplets that are periodically distributed in polymer networks on the scale of optical wavelengths. The alternating liquid crystal-rich and liquid crystal-depleted regions form the fringe planes of the grating, which can produce Bragg diffraction with a strong optical polarization resulting from the orientation ordering of the LC molecules in the droplets.
The resulting volume phase grating can exhibit very high diffraction efficiency, which can be controlled by the magnitude of the electric field applied across the film. When an electric field is applied to the grating via transparent electrodes, the natural orientation of the LC droplets can change, causing the refractive index modulation of the fringes to lower and the hologram diffraction efficiency to drop to very low levels. Typically, the electrodes are configured such that the applied electric field will be perpendicular to the substrates. In a number of embodiments, the electrodes are fabricated from indium tin oxide (ITO). In the OFF state with no electric field applied, the extraordinary axis of the liquid crystals generally aligns normal to the fringes. The grating thus exhibits high refractive index modulation and high diffraction efficiency for P-polarized light. When an electric field is applied to the HPDLC, the grating switches to the ON state wherein the extraordinary axes of the liquid crystal molecules align parallel to the applied field and hence perpendicular to the substrate. In the ON state, the grating exhibits lower refractive index modulation and lower diffraction efficiency for both S- and P-polarized light. Thus, the grating region no longer diffracts light. Each grating region can be divided into a multiplicity of grating elements such as for example a pixel matrix according to the function of the HPDLC device. Typically, the electrode on one substrate surface is uniform and continuous, while electrodes on the opposing substrate surface are patterned in accordance to the multiplicity of selectively switchable grating elements.
Typically, the SBG elements are switched clear in 30 μs with a longer relaxation time to switch ON. The diffraction efficiency of the device can be adjusted, by means of the applied voltage, over a continuous range. In many cases, the device exhibits near 100% efficiency with no voltage applied and essentially zero efficiency with a sufficiently high voltage applied. In certain types of HPDLC devices, magnetic fields can be used to control the LC orientation. In some HPDLC applications, phase separation of the LC material from the polymer can be accomplished to such a degree that no discernible droplet structure results. An SBG can also be used as a passive grating. In this mode, its chief benefit is a uniquely high refractive index modulation. SBGs can be used to provide transmission or reflection gratings for free space applications. SBGs can be implemented as waveguide devices in which the HPDLC forms either the waveguide core or an evanescently coupled layer in proximity to the waveguide. The substrates used to form the HPDLC cell provide a total internal reflection (TIR) light guiding structure. Light can be coupled out of the SBG when the switchable grating diffracts the light at an angle beyond the TIR condition.
In some embodiments, LC can be extracted or evacuated from the SBG to provide an evacuated Bragg grating (EBG). EBGs can be characterized as a surface relief grating (SRG) that has properties very similar to a Bragg grating due to the depth of the SRG structure (which is much greater than that practically achievable using surface etching and other conventional processes commonly used to fabricate SRGs). The LC can be extracted using a variety of different methods, including but not limited to flushing with isopropyl alcohol and solvents. In many embodiments, one of the transparent substrates of the SBG is removed, and the LC is extracted. In further embodiments, the removed substrate is replaced. The SRG can be at least partially backfilled with a material of higher or lower refractive index. Such gratings offer scope for tailoring the efficiency, angular/spectral response, polarization, and other properties to suit various waveguide applications.
Waveguides in accordance with various embodiments of the invention can include various grating configurations designed for specific purposes and functions. In many embodiments, the waveguide is designed to implement a grating configuration capable of preserving eyebox size while reducing lens size by effectively expanding the exit pupil of a collimating optical system. The exit pupil can be defined as a virtual aperture where the eye may be placed to receive the entire available field of view of the waveguide. As the eye moves beyond the exit pupil, light extracted from the waveguide can still be intercepted by the eye but with a progressively increasing loss of FOV. In some embodiments, the waveguide includes an input grating optically coupled to a light source, a fold grating for providing a first direction beam expansion, and an output grating for providing beam expansion in a second direction, which is typically orthogonal to the first direction, and beam extraction towards the eyebox. As can readily be appreciated, the grating configuration implemented in a waveguide architecture can depend on the specific requirements of a given application. In some embodiments, the grating configuration includes multiple fold gratings. In several embodiments, the grating configuration includes an input grating and a second grating for performing beam expansion and beam extraction simultaneously. The second grating can include gratings of different prescriptions, for propagating different portions of the field-of-view, arranged in separate overlapping grating layers or multiplexed in a single grating layer. Furthermore, various types of gratings and waveguide architectures can also be utilized.
In several embodiments, the gratings within each layer are designed to have different spectral and/or angular responses. For example, in many embodiments, different gratings across different grating layers are overlapped, or multiplexed, to provide an increase in spectral bandwidth. In some embodiments, a full color waveguide is implemented using three grating layers, each designed to operate in a different spectral band (red, green, and blue). In other embodiments, a full color waveguide is implemented using two grating layers, a red-green grating layer and a green-blue grating layer. As can readily be appreciated, such techniques can be implemented similarly for increasing angular bandwidth operation of the waveguide. In addition to the multiplexing of gratings across different grating layers, multiple gratings can be multiplexed within a single grating layer—e.g., multiple gratings can be superimposed within the same volume. In several embodiments, the waveguide includes at least one grating layer having two or more grating prescriptions multiplexed in the same volume. In further embodiments, the waveguide includes two grating layers, each layer having two grating prescriptions multiplexed in the same volume. Multiplexing two or more grating prescriptions within the same volume can be achieved using various fabrication techniques. In a number of embodiments, a multiplexed master grating is utilized with an exposure configuration to form a multiplexed grating. In many embodiments, a multiplexed grating is fabricated by sequentially exposing an optical recording material layer with two or more configurations of exposure light, where each configuration is designed to form a grating prescription. In some embodiments, a multiplexed grating is fabricated by exposing an optical recording material layer by alternating between or among two or more configurations of exposure light, where each configuration is designed to form a grating prescription. As can readily be appreciated, various techniques, including those well known in the art, can be used as appropriate to fabricate multiplexed gratings.
In many embodiments, the waveguide can incorporate at least one of: angle multiplexed gratings, color multiplexed gratings, fold gratings, dual interaction gratings, rolled K-vector gratings, crossed fold gratings, tessellated gratings, chirped gratings, gratings with spatially varying refractive index modulation, gratings having spatially varying grating thickness, gratings having spatially varying average refractive index, gratings with spatially varying refractive index modulation tensors, and gratings having spatially varying average refractive index tensors. In some embodiments, the waveguide can incorporate at least one of: a half wave plate, a quarter wave plate, an anti-reflection coating, a beam splitting layer, an alignment layer, a photochromic back layer for glare reduction, and louvre films for glare reduction. In several embodiments, the waveguide can support gratings providing separate optical paths for different polarizations. In various embodiments, the waveguide can support gratings providing separate optical paths for different spectral bandwidths. In a number of embodiments, the gratings can be HPDLC gratings, switching gratings recorded in HPDLC (such switchable Bragg Gratings), Bragg gratings recorded in holographic photopolymer, or surface relief gratings. In many embodiments, the waveguide operates in a monochrome band. In some embodiments, the waveguide operates in the green band. In several embodiments, waveguide layers operating in different spectral bands such as red, green, and blue (RGB) can be stacked to provide a three-layer waveguiding structure. In further embodiments, the layers are stacked with air gaps between the waveguide layers. In various embodiments, the waveguide layers operate in broader bands such as blue-green and green-red to provide two-waveguide layer solutions. In other embodiments, the gratings are color multiplexed to reduce the number of grating layers. Various types of gratings can be implemented. In some embodiments, at least one grating in each layer is a switchable grating.
Waveguides incorporating optical structures such as those discussed above can be implemented in a variety of different applications, including but not limited to waveguide displays. In various embodiments, the waveguide display is implemented with an eyebox of greater than 10 mm with an eye relief greater than 25 mm. In some embodiments, the waveguide display includes a waveguide with a thickness between 2.0-5.0 mm. In many embodiments, the waveguide display can provide an image field-of-view of at least 50° diagonal. In further embodiments, the waveguide display can provide an image field-of-view of at least 70° diagonal. The waveguide display can employ many different types of picture generation units (PGUs). In several embodiments, the PGU can be a reflective or transmissive spatial light modulator such as a liquid crystal on Silicon (LCoS) panel or a micro electromechanical system (MEMS) panel. In a number of embodiments, the PGU can be an emissive device such as an organic light emitting diode (OLED) panel. In some embodiments, an OLED display can have a luminance greater than 4000 nits and a resolution of 4 k×4 k pixels. In several embodiments, the waveguide can have an optical efficiency greater than 10% such that a greater than 400 nit image luminance can be provided using an OLED display of luminance 4000 nits. Waveguides implementing P-diffracting gratings (e.g., gratings with high efficiency for P-polarized light) typically have a waveguide efficiency of 5%-6.2%. Since P-diffracting or S-diffracting gratings can waste half of the light from an unpolarized source such as an OLED panel, many embodiments are directed towards waveguides capable of providing both S-diffracting and P-diffracting gratings to allow for an increase in the efficiency of the waveguide by up to a factor of two. In some embodiments, the S-diffracting and P-diffracting gratings are implemented in separate overlapping grating layers. Alternatively, a single grating can, under certain conditions, provide high efficiency for both p-polarized and s-polarized light. In several embodiments, the waveguide includes Bragg-like gratings produced by extracting LC from HPDLC gratings, such as those described above, to enable high S and P diffraction efficiency over certain wavelength and angle ranges for suitably chosen values of grating thickness (typically, in the range 2-5 μm).
Various embodiments in accordance with the invention include a number of eye tracking solutions and methods for measuring eye movement and for implementing eye trackers. In many embodiments, classical eye tracking technology based on projecting IR light into the user's eye and utilizing the primary Purkinje reflections (from the cornea and lens surfaces) and the pupil-masked retina reflection can be implemented. The general strategy is to track the relative motion of these images in order to establish a vector characterizing the point of regard. In some embodiments, eye tracking solutions and methods may track eye movements using just one of the Purkinje reflections. In some embodiments, tracking more than one of the reflections may allow more accurate determination of gaze vectors. The cornea, which has an aspheric shape of smaller radius than the eye-ball, can provide a reflection that tracks fairly well with angular motion until the reflected image falls off the edge of the cornea and onto the sclera. Most solutions rely on projecting IR light into the user's eye and tracking the reflections from the principal surfaces, e.g., at least one surface of the lens, cornea and retina. However, in some cases, there are challenges to introducing the image sensor and illuminator for efficient operation while avoiding obscuring the line of sight. Most eye tracker implementations in HMDs have employed flat beam splitters in front of the users' eyes and relatively large optics to image the reflections onto an imaging sensor. A flat beam splitter may be a substrate inclined at an angle to the user's eye which transmits light towards the eye and allows a portion of the light reflected from the eye to be directed onto a detector (e.g. an imaging sensor). The beam splitter must cover the eye box to intercept all reflected rays and also needs to be far enough from the eye to maintain adequate eye relief (e.g. greater than or equal to 15 mm), particularly if the waveguide-based display includes prescription spectacles. This results in quite a large beam splitter element. From basic geometrical optics, the detector lens which is used to form an image of the tracked feature of the eye on the detector will be of comparable size to the beam splitter. Hence, the beam splitter and lens assembly of this type may have an unacceptable weight and form factor especially for HMDs. Inevitably, there are tradeoffs between exit pupil, field of view, and ergonomics. The exit pupil is generally limited by either the beamsplitter size or the first lens of the imaging optics. In order to maximize the exit pupil, the imaging optics are positioned close to the beamsplitter, presenting vision obscuration and a safety hazard. Another challenge with eye trackers is the field of view, which is generally limited by the illumination scheme in combination with the geometry of the reflected images. The size of the corneal reflected angles would ordinarily require a large angular separation between the illumination and detection optical axes, making using corneal reflections over large FOVs very difficult. Ideally, the eye tracker should minimize the angle between the illumination and reflection beams. The temporal resolution of an eye tracker should be at least 60 Hz. However, 90-120 Hz is preferred. Direct imaging by miniature cameras is becoming more attractive as cameras get smaller, and their resolution increases. However, the latency incurred by the need to recognize and track eye features remains a significant processing bottleneck. From the optical and ergonomic perspective, providing a line-of-sight for a camera in an HMD is not trivial.
In eye tracking applications, the signatures to be recorded do not need to be images of eye features, such as pupil edges, but can be random structures, such as but not limited to speckle patterns (e.g. reflections from multiple surfaces and scatter from the optical media inside the eye). In some examples, the speckle patterns can be a statistic intensity variation arising from non-uniformities in the various eye media and/or surfaces. Alternatively, the speckle patterns may be introduced into the illumination beams to provide a statistical structured light pattern that may be used to track the eye. In many embodiments, the tracked signature has a strong spatio-temporal variation with gaze direction. Such eye trackers in accordance with various embodiments of the invention can be implemented in a variety of configurations. In some embodiments, the eye tracker provides an infrared illumination optical channel for delivering infrared illumination to the eye and an imaging or detection optical channel for forming an image (or recording a signature) of the eye at a detector. The optical channels can be implemented in many different ways, including but not limited to the use of waveguiding structures.
In some embodiments, the eye tracker waveguide solution may track the spatial location of the eye box of the user. Typical eye tracker systems may track only the angle of the user's eye in order to provide resolution correction to the user. In some examples, typical eye tracker systems may use the angle of the user's eye to locate the user's gaze. The display may implement a lower resolution image until the eye is fixated along a given direction in which case the display switches to a higher resolution image. However, in these instances, the eye tracker system would not track the spatial location of the eye box of the user. The spatial location of the eye box of the user may be used to provide color or image correction. Without limitation to any particular theory, as the eye rotates from a forward gaze direction, the center of the eye pupil translates spatially producing spatial variation.
A ‘rainbow’ effect in the waveguide may be caused by this spatial variation. The user's eye rotation may cause pupil translation which may vary the field of color differently with different eye box positions. Correcting for spatial variations in the color uniformity in the eye box may be advantageous. In some embodiments, the spatial variations in the color uniformity may be superimposed on the image which may cause the color to vary as the eye rotates. The perceived variation of color (or luminance and contrast) with angle can vary with position in the eye box. This variation may be a consequence of system aberrations that can arise in the picture generation optics (projection lens) and from the variation of diffraction efficiency with wavelength and angle within the waveguide. The color variation nonuniformity may also be different for the red, green blue components of a color image. Based on the prescriptions of the optical elements along the beam path from the microdisplay panel to the eyebox, the resulting spatial values of color nonuniformity at points in the eyebox can be determined using ray tracing. From basic colorimetry theory, the tristimulus values (which specify a color's lightness, hue and saturation, from which the chromaticity coordinates may be computed) the adjustment to bring the RGB chromaticity coordinates on target at each pupil point and for each viewing direction can be calculated. In some embodiments, this calculation can be performed in real-time following each gaze vector and pupil position update from the eye tracker. In many embodiments, it may be more computationally efficient to determine the complete set of color coordinate adjustments for each pupil position and viewing direction in an initial calibration of the eye tracker, the adjustments being stored in a look-up table (LUT). A separate LUT can be provided for each of the red, green and blue color components, with the corrections for each angular direction being applied to individual pixels of the microdisplay. In some embodiments, the corrections can be computed for submatrices of the input image pixel array using eye tracker data sampled at lower angular and spatial frequencies. The correction for individual pixels can then be obtained using interpolation. This procedure can avoid the processing latency that may result from having to perform colorimetric calculations in real-time. In some embodiments, predictive eye tracking algorithms can be used in association with color correction to reduce latency. Various criteria can be used to characterize color correction. For example, one criteria may be based on adjusting the viewed image white point variation across the field of view to some predefined range of values. Another approach may involve adjusting color space parameters to achieve some predefined color gamut variation across the field of view.
Referring back to
In
The output rays 112 are reflected off the front surface of the cornea 113 and the retina 114 as return light 104 and 115, respectively. The corneal and retinal image light can be coupled back into the waveguide 101 by an active element of the imaging grating 109, which can be switched one element at a time, into a TIR path, illustrated by rays such as 116. The light 116 is deflected into a ray path 117 toward the detector 106 by the output grating 110, which can be switched one element at a time. In some embodiments, the detector 106 is a two-dimensional array detector. Other types of detectors can also be used. For example, linear arrays and analogue devices such as but not limited to position sensing detectors can be used. In many embodiments, the detector 106 reads out the image signal in synchronism with the switching of the elements. The detector 106 can be connected to an image processing apparatus for determining at least one spatio-temporal characteristic of an eye movement. The image processor, which is not illustrated, can detect pre-defined features of the backscattered signals from the cornea and retina. For example, the image processor may be used to determine the centroid of an eye feature such as the pupil. Other trackable features of the eye will be well known to those skilled in arts of eye tracker design and visual optics.
Although
The gratings may be surface relief gratings. However, such gratings may be inferior to Bragg gratings in terms of their optical efficiency and angular/wavelength selectivity. Although the invention is discussed in relation to transmission gratings, it should be apparent to those skilled in the art that equivalent embodiments using reflection gratings can also be implemented.
In addressing the color non-uniformity problem with respect to waveguide applications, different systems and methods can be implemented as appropriate depending on the specific requirements of a given application. Color non-uniformity can arise from various sources. For example, in full-color RGB waveguides, one possible source results from non-matching, non-uniform profiles for each of the red, green, and blue outputs. The superimposition of the three output forms an output profile that is spectrally non-uniform. In principle, color non-uniformity can be pre-compensated or pre-corrected for during processing by altering the values and characteristics of the input image light such that any effect or transformation performed by the gratings will result in the desired original image. For example, in full-color RGB waveguides, the proportions of red, green, and blue light in the original image light can be altered such that the effects of the gratings on the final output image are compensated. However, in many cases, color non-uniformity profiles can vary based on the position of the user's pupil and/or the angle at which the user views the image. Such variations can be difficult or impossible to correct for across all viewing conditions during processing. One class of alternative solutions includes performing corrections in real-time based on the user's viewing condition. In such embodiments, a system can be implemented with sensors to determine the user's viewing condition, which the system can then utilize to perform the appropriate color correction to the input image light in order to mitigate color non-uniformity in the output.
Real-time color correction processes in accordance with various embodiments of the invention can be implemented with a variety of different systems. In many embodiments, the system is implemented as a waveguide display and includes an optical engine, a waveguide, and at least one eye tracker. Optical engines can include various illumination schemes. In some embodiments, the optical engine includes a compact laser illumination delivery solution. In other embodiments, the optical engine implements LED illumination. In several embodiments, the optical engine includes a PGU. Utilizing information from the eye tracker, the system can be configured to determine the alterations to the input image light required to achieve the desired compensation. In such embodiments, the system can include a controller or electronics for processing information from the eye tracker to provide a signal to the optical engine relating to the appropriate corrections needed to be made.
In some embodiments, the infrared detector 408 and/or controller 415 may track the spatial location of the eye box of the user. Typical eye tracker systems may track only the angle of the user's eye in order to provide resolution correction to the user. In some examples, typical eye tracker systems may use the angle of the user's eye to locate the user's gaze. The display may implement a lower resolution image until the eye fixates along a given viewing direction in which case the display switches to a higher resolution image. However, in these instances, the eye tracker system would not track the spatial location of the eye box of the user. In some embodiments, the controller 415 may use the spatial location of the eye box of the user to provide color or image correction. Without limitation to any particular theory, as the eye rotates from a forward gaze direction, the eye translates spatially producing spatial variation. A ‘rainbow’ effect in the waveguide may be caused by this spatial variation. The user's eye rotation may cause pupil translation which may vary the field of color differently with different eye box positions. Correcting for spatial variations in the color uniformity in the eye box may be advantageous. In some embodiments, the spatial variations in the color uniformity may be superimposed on the image which may cause the color to vary as the eye rotates.
As discussed above, information from the eye tracker can be utilized to determine in real-time the compensations to be performed to the input image light. In some embodiments, information from the eye tracker is used to determine the user's viewing condition. Information capable of determining the user's viewing condition can include but are not limited to eye relief information, location of the user's gaze, location of the eye relative to the user's head, waveguide, eye tracker, or any other known object, and location of features (pupil, iris, etc.) of the user's eye. Depending on the user's viewing condition, the system can perform the appropriate compensation to the input image light accordingly. In a number of embodiments, the compensation is performed by configuring the optical engine to output the desired, compensated light. In some embodiments, an electrically controllable component is implemented to modify the light outputted from the optical engine to perform the desired compensation. The rate at which compensations and corrections are performed can be limited by several factors, including but not limited to mechanical or technical limitations of the system. In some cases, the compensations are performed at a rate proportional to the sampling rate of the eye tracker implementation. In several embodiments, the rate at which the compensations are performed is limited by the computational power of the system. As can readily be appreciated, the rates at which compensations are performed can vary and can depend on the specific requirements of a given application.
In many embodiments, information from the eye tracker is used in conjunction with a lookup table to control the output of the optical engine—e.g., the input image light. In such embodiments, the process implemented can include a pre-calibration step to map out and determine the waveguide's characteristics and transformation effects on the input light. The information from the pre-calibration step can be utilized to create a lookup table detailing the compensations necessary for a given viewing condition for a given waveguide. As can readily be appreciated, a lookup table can be formed for and associated to each individual waveguide. In many embodiments, the lookup table includes a finite list of viewing conditions. Such embodiments can be implemented by discretizing and classifying all possible viewing conditions using a predetermined rule. For example, each possible viewing conditions can be associated with its most comparable (as characterized and identified by a predetermined rule) viewing condition within the finite list. As can readily be appreciated, the classification and discretization process can be implemented as appropriate depending on the specific requirements of a given application.
A compensation factor can be determined (505) using the user's viewing condition. The compensation factor may include a color correction or an image correction as described above. As described above, the compensations to be made on the input image can be determined using a variety of different methods, including but not limited to the use of a lookup table in conjunction with the viewing condition information. The determined compensation can be applied (506) to an input image light. In many implementations, the compensations are applied to input image light originating from an optical engine. Light sources used in the optical engine can include but are not limited to laser and LED sources. In some display applications, the compensated input image light is directed towards and coupled into a waveguide for propagation towards a user's eye. As can readily be appreciated, the compensations can be determined and applied at a predetermined frequency that can depend on a number of factors, including but not limited to the operating frequency of the detector. For example, many embodiments of the invention are capable of performing the compensations at frequencies of up to 120 Hz. In many embodiments, the compensations are performed at a frequency of 60 Hz. In some embodiments, the compensations are performed at 90 Hz.
Although
While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.
This current application is a U.S. National Stage Patent Application of PCT Patent Application No. PCT/US2021/038542 entitled “Systems and Methods for Real-Time Color Correction Of Waveguide Based Displays,” filed Jun. 22, 2021, which claims the benefit of priority to U.S. Provisional Patent Application No. 63/042,398 entitled “Systems and Methods for Real-Time Color Correction for Waveguide Displays and Related Applications,” filed Jun. 22, 2020. The disclosure of the provisional patent application is hereby incorporated by reference in its entirety for all purposes.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2021/038542 | 6/22/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63042398 | Jun 2020 | US |
