Patent Grant
11461961
Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR,” scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR,” scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user.
Despite the progress made in these display technologies, there is a need in the art for improved methods, systems, and devices related to augmented reality systems, particularly, display systems.
The present disclosure relates generally to techniques for improving optical systems in varying ambient light conditions. More particularly, embodiments of the present disclosure provide systems and methods for operating an augmented reality (AR) device comprising a dimming element. Although the present invention is described in reference to an AR device, the disclosure is applicable to a variety of applications in computer vision and image display systems.
A summary of the invention is provided below in reference to several examples. As used below, any reference to a series of examples is to be understood as a reference to each of those examples disjunctively (e.g., “Examples 1-4” is to be understood as “Examples 1, 2, 3, or 4”).
Example 1 is a method of operating an optical system, the method comprising: receiving, at the optical system, light associated with a world object; projecting virtual image light onto an eyepiece; determining a portion of a system field of view of the optical system to be at least partially dimmed based on information detected by the optical system; and adjusting a dimmer to reduce an intensity of the light associated with the world object in the portion of the system field of view.
Example 2 is the method of example(s) 1, wherein the optical system comprises a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.
Example 3 is the method of example(s) 1-2, wherein the light information includes a plurality of spatially-resolved light values.
Example 4 is the method of example(s) 1-3, wherein the light information includes a global light value.
Example 5 is the method of example(s) 1-4, wherein the optical system comprises an eye tracker configured to detect gaze information corresponding to an eye of a user of the optical system, wherein the detected information includes the gaze information.
Example 6 is the method of example(s) 1-5, wherein the gaze information includes a pixel location that intersects with a gaze vector of the eye of the user.
Example 7 is the method of example(s) 1-6, wherein the gaze information includes one or more of a pupil position of the eye of the user, a center of rotation of the eye of the user, a pupil size of the eye of the user, a pupil diameter of the eye of the user, and cone and rod locations of the eye of the user.
Example 8 is the method of example(s) 1-7, further comprising: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.
Example 9 is the method of example(s) 1-8, wherein the image information includes a plurality of spatially-resolved image brightness values.
Example 10 is the method of example(s) 1-9, wherein the image information includes a global image brightness value.
Example 11 is the method of example(s) 1-10, further comprising: determining a plurality of spatially-resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.
Example 12 is the method of example(s) 1-11, wherein the dimmer comprises a plurality of pixels.
Example 13 is the method of example(s) 1-12, wherein the dimmer is adjusted to completely block the intensity of the light associated with the world object in all of the system field of view.
Example 14 is the method of example(s) 1-13, further comprising: adjusting a brightness associated with the virtual image light.
Example 15 is the method of example(s) 1-14, wherein the virtual image light is characterized by an image field of view, and wherein the image field of view is equal to the system field of view.
Example 16 is the method of example(s) 1-15, wherein determining a portion of a system field of view of the optical system to be at least partially dimmed is based at least partly on the at least one world object.
Example 17 is the method of example(s) 1-16, wherein determining a portion of a system field of view of the optical system to be at least partially dimmed is based at least partly on at least one object included in the virtual image.
Example 18 is a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving, at an optical system, light associated with a world object; projecting virtual image light onto an eyepiece; determining a portion of a system field of view of the optical system to be at least partially dimmed based on information detected by the optical system; and adjusting a dimmer to reduce an intensity of the light associated with the world object in the portion of the system field of view.
Example 19 is the non-transitory computer-readable medium of example(s) 18, wherein the optical system comprises a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.
Example 20 is the non-transitory computer-readable medium of example(s) 19, wherein the light information includes a plurality of spatially-resolved light values.
Example 21 is the non-transitory computer-readable medium of example(s) 19, wherein the light information includes a global light value.
Example 22 is the non-transitory computer-readable medium of example(s) 18, wherein the optical system comprises an eye tracker configured to detect gaze information corresponding to an eye of a user of the optical system, wherein the detected information includes the gaze information.
Example 23 is the non-transitory computer-readable medium of example(s) 22, wherein the gaze information includes a pixel location that intersects with a gaze vector of the eye of the user.
Example 24 is the non-transitory computer-readable medium of example(s) 22, wherein the gaze information includes one or more of a pupil position of the eye of the user, a center of rotation of the eye of the user, a pupil size of the eye of the user, a pupil diameter of the eye of the user, and cone and rod locations of the eye of the user.
Example 25 is the non-transitory computer-readable medium of example(s) 18, wherein the operations further comprise: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.
Example 26 is the non-transitory computer-readable medium of example(s) 25, wherein the image information includes a plurality of spatially-resolved image brightness values.
Example 27 is the non-transitory computer-readable medium of example(s) 25, wherein the image information includes a global image brightness value.
Example 28 is the non-transitory computer-readable medium of example(s) 18, wherein the operations further comprise: determining a plurality of spatially-resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.
Example 29 is the non-transitory computer-readable medium of example(s) 18, wherein the dimmer comprises a plurality of pixels.
Example 30 is the non-transitory computer-readable medium of example(s) 18, wherein the dimmer is adjusted to completely block the intensity of the light associated with the world object in all of the system field of view.
Example 31 is the non-transitory computer-readable medium of example(s) 18, wherein the operations further comprise: adjusting a brightness associated with the virtual image light.
Example 32 is the non-transitory computer-readable medium of example(s) 18, wherein the virtual image light is characterized by an image field of view, and wherein the image field of view is equal to the system field of view.
Example 33 is an optical system comprising: a projector configured to project virtual image light onto an eyepiece; a dimmer configured to dim light associated with a world object; a processor communicatively coupled to the projector and the dimmer, wherein the processor is configured to perform operations comprising: determining a portion of a system field of view of the optical system to be at least partially dimmed based on information detected by the optical system; and adjusting the dimmer to reduce an intensity of the light associated with the world object in the portion of the system field of view.
Example 34 is the optical system of example(s) 33, further comprising: a light sensor configured to detect light information corresponding to the light associated with the world object, wherein the detected information includes the light information.
Example 35 is the optical system of example(s) 34, wherein the light information includes a plurality of spatially-resolved light values.
Example 36 is the optical system of example(s) 34, wherein the light information includes a global light value.
Example 37 is the optical system of example(s) 33, further comprising: an eye tracker configured to detect gaze information corresponding to an eye of a user of the optical system, wherein the detected information includes the gaze information.
Example 38 is the optical system of example(s) 37, wherein the gaze information includes a pixel location that intersects with a gaze vector of the eye of the user.
Example 39 is the optical system of example(s) 37, wherein the gaze information includes one or more of a pupil position of the eye of the user, a center of rotation of the eye of the user, a pupil size of the eye of the user, a pupil diameter of the eye of the user, and cone and rod locations of the eye of the user.
Example 40 is the optical system of example(s) 33, wherein the operations further comprise: detecting image information corresponding to the virtual image light, wherein the detected information includes the image information.
Example 41 is the optical system of example(s) 40, wherein the image information includes a plurality of spatially-resolved image brightness values.
Example 42 is the optical system of example(s) 40, wherein the image information includes a global image brightness value.
Example 43 is the optical system of example(s) 33, wherein the operations further comprise: determining a plurality of spatially-resolved dimming values for the portion of the system field of view based on the detected information, wherein the dimmer is adjusted according to the plurality of dimming values.
Example 44 is the optical system of example(s) 33, wherein the dimmer comprises a plurality of pixels.
Example 45 is the optical system of example(s) 33, wherein the dimmer is adjusted to completely block the intensity of the light associated with the world object in all of the system field of view.
Example 46 is the optical system of example(s) 33, wherein the operations further comprise: adjusting a brightness associated with the virtual image light.
Example 47 is the optical system of example(s) 33, wherein the virtual image light is characterized by an image field of view, and wherein the image field of view is equal to the system field of view.
Example 48 is an optical system comprising: a frame configured to be worn about a head of a user of the optical system; a dimming component carried by the frame and configured to be positioned between an eye of the user and an environment of the user; an eye tracker configured to monitor a positioning of the eye of the user; and control circuitry communicatively coupled to the dimming component and the eye tracker, the control circuitry configured to: receive data from the eye tracker; determine, based on the data received from the eye tracker, a location along an optical axis of the eye of the user at which a particular anatomical region of the eye of the user is positioned; identify one or more points in three-dimensional space located within the environment of the user; and for each of the one or more identified points within the environment of the user: identify a set of one or more pixels of the dimming component based at least in part on the determined location of the particular anatomical region of the eye of the user and the respective point in three-dimensional space located within the environment of the user; and control the dimming component to dim the identified set of one or more pixels.
Example 49 is the optical system of example(s) 48, wherein the particular anatomical region of the eye of the user comprises a center of rotation of the eye of the user.
Example 50 is the optical system of example(s) 48, wherein the particular anatomical region of the eye of the user comprises a center of a pupil of the eye of the user.
Example 51 is the optical system of example(s) 48, further comprising: a projector configured to emit light representing virtual content; and a waveguide carried by the frame and configured to be positioned between the eye of the user and the dimming component, wherein the waveguide is configured to receive and direct light from the projector to the eye of the user.
Example 52 is the optical system of example(s) 51, wherein the control circuitry is communicatively coupled to the projector, the control circuitry further configured to control the projector to emit light representing one or more pixels of virtual content.
Example 53 is the optical system of example(s) 52, wherein the one or more points in three-dimensional space located within the environment of the user correspond to one or more locations in three-dimensional space at which the one or more pixels of virtual content are to be perceived by the user, respectively.
Example 54 is the optical system of example(s) 52, wherein the one or more pixels of virtual content comprise a plurality of pixels of a virtual object.
Example 55 is the optical system of example(s) 54, wherein the one or more points in three-dimensional space located within the environment of the user correspond to one or more locations in three-dimensional space at which one or more pixels of a virtual shadow associated with the virtual object is to be perceived by the user, respectively.
Example 56 is the optical system of example(s) 48, wherein the one or more points in three-dimensional space located within the environment of the user correspond to one or more points in three-dimensional space physically occupied by a real world object in the environment of the user.
Example 57 is the optical system of example(s) 48, wherein to identify the set of one or more pixels of the dimming component, the control circuitry is configured to: cast a set of one or more rays from the respective point in three-dimensional space located within the environment of the user to the determined location of the particular anatomical region of the eye of the user; and identify a set of one or more points of intersection between the set of one or more rays and the dimming component.
Example 58 is the optical system of example(s) 48, wherein the dimming component is curved in shape.
Example 59 is the optical system of example(s) 48, wherein the control circuitry is further configured to determine a set of one or more dimming values for the identified set of one or more pixels of the dimming component, respectively, and wherein the control circuitry is configured to control the dimming component to dim the identified set of one or more pixels in accordance with the determined set of one or more dimming values.
Example 60 is the optical system of example(s) 59, wherein the control circuitry is further configured to determine one or more characteristics of the eye of the user based on the data received from the eye tracker, and wherein the control circuitry is configured to determine the set of one or more dimming values for the identified set of one or more pixels of the dimming component, respectively, based at least in part on the one or more determined characteristics of the eye of the user.
Example 61 is the optical system of example(s) 60, wherein the one or more characteristics of the eye of the user include one or more of a pupil size of the eye of the user, a pupil diameter of the eye of the user, cone and rod locations of the eye of the user, and an accommodative state of a lens of the eye of the user.
Example 62 is the optical system of example(s) 61, wherein the control circuitry is configured to identify the set of one or more pixels of the dimming component based at least in part on the one or more determined characteristics of the eye of the user.
Example 63 is the optical system of example(s) 59 further comprising: a projector communicatively coupled to the control circuitry and configured to emit light representing virtual content; and a waveguide carried by the frame and configured to be positioned between the eye of the user and the dimming component, wherein the waveguide is configured to receive and direct light from the projector to the eye of the user, wherein the control circuitry is further configured to control the projector to emit light representing one or more pixels of virtual content at one or more levels of brightness, respectively, and wherein the control circuitry is configured to determine the set of one or more dimming values for the identified set of one or more pixels of the dimming component, respectively, based at least in part on the one or more levels of brightness of the one or more pixels of virtual content.
Example 64 is the optical system of example(s) 63, wherein the control circuitry is configured to determine the set of one or more dimming values for the identified set of one or more pixels of the dimming component, respectively, based at least in part on one or more of a predetermined contrast and a predetermined level of visibility specified for the virtual content.
Example 65 is the optical system of example(s) 63, wherein the virtual content comprises a virtual object, and wherein the control circuitry is configured to identify the set of one or more pixels of the dimming component based at least in part on one or more characteristics of the virtual object.
Example 66 is the optical system of example(s) 65, wherein the one or more characteristics of the virtual object include one or more of a size of the virtual object, a shape of the virtual object, a position in the environment of the user at which the virtual object is to be perceived by the user, and a depth at which the virtual object is to be perceived by the user.
Example 67 is the optical system of example(s) 59 further comprising: an optical sensor communicatively coupled to the control circuitry and configured to monitor one or more levels of brightness of light associated with one or more portions of the environment of the user, respectively, and wherein the control circuitry is configured to determine the set of one or more dimming values for the identified set of one or more pixels of the dimming component, respectively, based at least in part on the one or more levels of brightness associated with the one or more portions of the environment of the user.
Example 68 is the optical system of example(s) 67, wherein the optical sensor comprises a camera.
Example 69 is the optical system of example(s) 67, wherein the optical sensor comprises one or more photodiodes.
Example 70 is the optical system of example(s) 67 further comprising: a projector communicatively coupled to the control circuitry and configured to emit light representing virtual content; and a waveguide carried by the frame and configured to be positioned between the eye of the user and the dimming component, wherein the waveguide is configured to receive and direct light from the projector to the eye of the user, and wherein the control circuitry is further configured to control the projector to emit light representing one or more pixels of virtual content.
Example 71 is the optical system of example(s) 70, wherein the virtual content comprises a virtual object, and wherein the one or more portions of the environment of the user with which the one or more levels of brightness are associated include a particular portion of the environment of the user that is to be perceived by the user as occluded by the virtual object.
Example 72 is the optical system of example(s) 70, wherein the control circuitry is further configured to control the projector to emit light representing one or more pixels of virtual content based at least in part on the one or more levels of brightness associated with the one or more portions of the environment of the user.
Example 73 is the optical system of example(s) 48, wherein the eye tracker is configured to monitor the positioning of the eye of the user relative to the dimming component.
Example 74 is an optical system comprising: a frame configured to be worn about a head of a user of the optical system; a left dimming component carried by the frame and configured to be positioned between a left eye of the user and an environment of the user; a right dimming component carried by the frame and configured to be positioned between a right eye of the user and the environment of the user; and control circuitry communicatively coupled to the left and right dimming components, the control circuitry configured to: identify one or more points in three-dimensional space located within the environment of the user; and for each of the one or more identified points within the environment of the user: identify a set of one or more pixels of the left dimming component based at least in part on the respective point in three-dimensional space located within the environment of the user; identify a set of one or more pixels of the right dimming component based at least in part on the respective point in three-dimensional space located within the environment of the user; control the left dimming component to dim the identified set of one or more pixels of the left dimming component; and control the right dimming component to dim the identified set of one or more pixels of the right dimming component.
Example 75 is the optical system of example(s) 74 further comprising: a left eye tracker communicatively coupled to the control circuitry and configured to monitor a positioning of the left eye of the user; and a right eye tracker communicatively coupled to the control circuitry and configured to monitor a positioning of the left eye of the user; wherein the control circuitry is further configured to: receive data from the left and right eye trackers; determine, based on the data received from the left eye tracker, a location along an optical axis of the left eye of the user at which a particular anatomical region of the left eye of the user is positioned; and determine, based on the data received from the right eye tracker, a location along an optical axis of the right eye of the user at which a particular anatomical region of the right eye of the user is positioned;
Example 76 is the optical system of example(s) 75, wherein the control circuitry is configured to: identify the set of one or more pixels of the left dimming component based at least in part on the determined location of the particular anatomical region of the left eye of the user and the respective point in three-dimensional space located within the environment of the user; and identify the set of one or more pixels of the right dimming component based at least in part on the determined location of the particular anatomical region of the right eye of the user and the respective point in three-dimensional space located within the environment of the user.
Numerous benefits are achieved by way of the present disclosure over conventional techniques. For example, augmented reality (AR) devices described herein may be used in varying light levels, from dark indoors to bright outdoors, by globally dimming and/or selectively dimming the ambient light reaching the user's eyes. Embodiments of the present invention allow for AR and virtual reality (VR) capabilities in a single device by using the pixelated dimmer to attenuate the world light by greater than 99%. Embodiments of the present invention also mitigate vergence accommodation conflict using a variable focal element with discrete or continuous variable depth plane switching technologies. Embodiments of the present invention improve the battery life of the AR device by optimizing the projector brightness based on the amount of detected ambient light. Other benefits of the present disclosure will be readily apparent to those skilled in the art.
An ongoing technical challenge with optical see through (OST) augmented reality (AR) devices is the variation in the opacity and/or visibility of the virtual content under varying ambient light conditions. The problem worsens in extreme lighting conditions such as a completely dark room or outside in full bright sunlight. Embodiments of the present invention solve these and other problems by dimming the world light at different spatial locations within the field of view of the AR device. The portion of the field of view to which dimming is applied and the amount of dimming that is applied are each determined based on various information detected by the AR device. This information may include detected ambient light, detected gaze information, and/or the detected brightness of the virtual content being projected. The functionality of the AR device is further improved by detecting a direction associated with the ambient light by, for example, detecting a plurality of spatially-resolved light values. This allows the AR device to improve its battery life by only dimming the portions of the field of view in which dimming is needed and/or increasing the projector brightness in certain portions of the field of view. Accordingly, embodiments of the present invention enable usage of the AR device in a much wider variety of conditions than traditionally possible.
During operation, dynamic dimmer 203 may be adjusted to reduce an intensity of a world light 232 associated with world objects 230 impinging on dynamic dimmer 203, thereby producing a dimmed area 236 within the system field of view. Dimmed area 236 may be a portion or subset of the system field of view, and may be partially or completely dimmed. Dynamic dimmer 203 may be adjusted according to a plurality of spatially-resolved dimming values for dimmed area 236. Furthermore, during operation of AR device 200, projector 214 may project a virtual image light 222 (i.e., light associated with virtual content) onto eyepiece 202 which may be observed by the user along with world light 232.
Projecting virtual image light 222 onto eyepiece 202 may cause a light field (i.e., an angular representation of virtual content) to be projected onto the user's retina in a manner such that the user perceives the corresponding virtual content as being positioned at some location within the user's environment. For example, virtual image light 222 outcoupled by eyepiece 202 may cause the user to perceive character 202-1 as being positioned at a first virtual depth plane 210-1 and statue 202-2 as being positioned at a second virtual depth plane 210-2. The user perceives the virtual content along with world light 232 corresponding to one or more world objects 230, such as platform 120.
In some embodiments, AR device 200 may include an ambient light sensor 234 configured to detect world light 232. Ambient light sensor 234 may be positioned such that world light 232 detected by ambient light sensor 234 is similar to and/or representative of world light 232 that impinges on dynamic dimmer 203 and/or eyepiece 202. In some embodiments, ambient light sensor 234 may be configured to detect a plurality of spatially-resolved light values corresponding to different pixels of dynamic dimmer 203. In these embodiments, ambient light sensor 234 may, for example, correspond to an imaging sensor (e.g., CMOS, CCD, etc.) or a plurality of photodiodes (e.g., in an array or another spatially-distributed arrangement). In some embodiments, or in the same embodiments, ambient light sensor 234 may be configured to detect a global light value corresponding to an average light intensity or a single light intensity of world light 232. In these embodiments, ambient light sensor 234 may, for example, correspond to a set of one or more photodiodes. Other possibilities are contemplated.
Some or all of the components of AR device 300 may be head mounted such that projected images may be viewed by a user. In one particular implementation, all of the components of AR device 300 shown in
Processing module 350 may include a processor 352 and an associated digital memory 356, such as non-volatile memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data. The data may include data captured from sensors (which may be, e.g., operatively coupled to AR device 300) or otherwise attached to a user, such as cameras 306, ambient light sensor 334, eye trackers 340, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros. For example, processing module 350 may receive image(s) 320 from cameras 306. Specifically, processing module 350 may receive left front image(s) 320A from left front-facing world camera 306A, right front image(s) 320B from right front-facing world camera 306B, left side image(s) 320C from left side-facing world camera 306C, and right side image(s) 320D from right side-facing world camera 306D. In some embodiments, image(s) 320 may include a single image, a pair of images, a video comprising a stream of images, a video comprising a stream of paired images, and the like. Image(s) 320 may be periodically generated and sent to processing module 350 while AR device 300 is powered on, or may be generated in response to an instruction sent by processing module 350 to one or more of the cameras. As another example, processing module 350 may receive light information from ambient light sensor 334. In some embodiments, some or all of the functionality of ambient light sensor 334 may be provided by way of one or more of world cameras 306A-306D. As another example, processing module 350 may receive gaze information from one or both of eye trackers 340. As another example, processing module 350 may receive image information (e.g., image brightness values) from one or both of projectors 314.
Eyepieces 302A and 302B may comprise transparent or semi-transparent waveguides configured to direct light from projectors 314A and 314B, respectively. Specifically, processing module 350 may cause left projector 314A to output a left virtual image light 322A onto left eyepiece 302A (causing a corresponding light field associated with left virtual image light 322A to be projected onto the user's retina), and may cause right projector 314B to output a right virtual image light 322B onto right eyepiece 302B (causing a corresponding light field associated with right virtual image light 322B to be projected onto the user's retina). In some embodiments, each of eyepieces 302 may comprise a plurality of waveguides corresponding to different colors and/or different depth planes. In some embodiments, dynamic dimmers 303 may be coupled to and/or integrated with eyepieces 302. For example, one of dynamic dimmers 303 may be incorporated into a multi-layer eyepiece and may form one or more layers that make up one of eyepieces 302.
Cameras 306A and 306B may be positioned to capture images that substantially overlap with the field of view of a user's left and right eyes, respectively. Accordingly, placement of cameras 306 may be near a user's eyes but not so near as to obscure the user's field of view. Alternatively or additionally, cameras 306A and 306B may be positioned so as to align with the incoupling locations of virtual image light 322A and 322B, respectively. Cameras 306C and 306D may be positioned to capture images to the side of a user, e.g., in a user's peripheral vision or outside the user's peripheral vision. Image(s) 320C and 320D captured using cameras 306C and 306D need not necessarily overlap with image(s) 320A and 320B captured using cameras 306A and 306B.
One or more components of AR device 300 may be similar to one or more components described in reference to
At step 402, light (e.g., world light 232) associated with a world object (e.g., world objects 230) is received at the optical system. The world object may be any number of real-world objects, such as a tree, a person, a house, a building, the sun, etc., that is viewed by a user of the optical system. In some embodiments, the light associated with the world object is first received by a dynamic dimmer (e.g., dynamic dimmer 203 or 303) or by an external cosmetic lens of the optical system. In some embodiments, the light associated with the world object is considered to be received at the optical system when the light reaches one or more components of the optical system (e.g., when the light reaches the dynamic dimmer).
At step 404, virtual image light (e.g., virtual image light 222 or 322) is projected onto an eyepiece (e.g., eyepiece 202 or 302). The virtual image light may be projected onto the eyepiece by a projector (e.g., projector 214 or 314) of the optical system. The virtual image light may correspond to a single image, a pair of images, a video comprising a stream of images, a video comprising a stream of paired images, and the like. In some embodiments, the virtual image light is considered to be projected onto the eyepiece when any light associated with the virtual image light reaches the eyepiece. In some embodiments, projecting the virtual image light onto the eyepiece causes a light field (i.e., an angular representation of virtual content) to be projected onto the user's retina in a manner such that the user perceives the corresponding virtual content as being positioned at some location within the user's environment.
During steps 406, 408, and 410, information may be detected by the optical system using, for example, one or more sensors of the optical system. At step 406, light information corresponding to the light associated with the world object is detected. The light information may be detected using a light sensor (e.g., ambient light sensor 234 or 334) mounted to the optical system. In some embodiments, the light information includes a plurality of spatially-resolved light values. Each of the plurality of spatially-resolved light values may correspond to a two-dimensional position within the system field of view. For example, each of the light values may be associated with a pixel of the dynamic dimmer. In other embodiments, or in the same embodiments, the light information may include a global light value. The global light value may be associated with the entire system field of view (e.g., an average light value of light impinging on all pixels of the dynamic dimmer).
At step 408, gaze information corresponding to an eye of a user of the optical system is detected. The gaze information may be detected using an eye tracker (e.g., eye tracker 240 or 340) mounted to the optical system. In some embodiments, the gaze information includes a gaze vector (e.g., gaze vector 238) of the eye of the user. In some embodiments, the gaze information includes one or more of a pupil position of the eye of the user, a center of rotation of the eye of the user, a pupil size of the eye of the user, a pupil diameter of the eye of the user, and cone and rod locations of the eye of the user. The gaze vector may be determined based on one or more components of the gaze information, such as the pupil position, the center of rotation of the eye, the pupil size, the pupil diameter, and/or the cone and rod locations. When the gaze vector is determined based on the cone and rod locations, it may further be determined based on the light information (e.g., the global light value) so as to determine an origin of the gaze vector within a retinal layer of the eye containing the cone and rod locations. In some embodiments, the gaze information includes a pixel or group of pixels of the dynamic dimmer at which the gaze vector intersects with the dynamic dimmer.
At step 410, image information corresponding to the virtual image light (e.g., virtual image light 222 or 322) projected by the projector onto the eyepiece is detected. The image information may be detected by the projector, by a processor (e.g., processor 352), or by a separate light sensor. In some embodiments, the image information includes one or more locations within the dynamic dimmer through which the user perceives the virtual content when the user observes the virtual image light. In some embodiments, the image information includes a plurality of spatially-resolved image brightness values (e.g., brightness of the perceived virtual content). For example, each of the image brightness values may be associated with a pixel of the eyepiece or of the dynamic dimmer. In one particular implementation, when the processor sends instructions to the projector to project the virtual image light onto the eyepiece, the processor may determine, based on the instructions, the spatially-resolved image brightness values. In another particular implementation, when the projector receives the instructions from the processor to project the virtual image light onto the eyepiece, the projector sends the spatially-resolved image brightness values to the processor. In another particular implementation, a light sensor positioned on or near the eyepiece detects and sends the spatially-resolved image brightness values to the processor. In other embodiments, or in the same embodiments, the image information includes a global image brightness value. The global image brightness value may be associated with the entire system field of view (e.g., an average image brightness value of all of the virtual image light).
At step 412, a portion of the system field of view to be at least partially dimmed is determined based on the detected information. The detected information may include the light information detected during step 406, the gaze information detected during step 408, and/or the image information detected during step 410. In some embodiments, the portion of the system field of view is equal to the entire system field of view. In various embodiments, the portion of the system field of view may be equal to 1%, 5%, 10%, 25%, 50%, or 75%, etc., of the system field of view. In some embodiments, the different types of information may be weighted differently in determining the portion to be at least partially dimmed. For example, gaze information, when available, may be weighted more heavily in determining the portion to be at least partially dimmed than light information and image information. In one particular implementation, each type of information may independently be used to determine a different portion of the system field of view to be at least partially dimmed, and subsequently the different portions may be combined into a single portion using an AND or an OR operation.
In some embodiments, the information used to determine a portion of the system field of view to be at least partially dimmed includes information associated with one or more objects that are presented within the virtual content. For example, the virtual content may include text, navigational indicators (e.g., arrows), and/or other content. The portion of the field of view in which such content is to be presented, and/or the field of view proximal to the content, can be dimmed such that the user can more easily read perceive and understand the content, and distinguish the content from world object(s). The dimmer can selectively dim one or more pixels and/or zone(s) of pixels, or enhance viewing of the content. In one example, a section of the lower portion of the field of view can be selectively and dynamically dimmed to make is easier for the user to see directional (e.g., navigation) arrows, text messages, and so forth. Such dimming may be performed while the content is being displayed in response to a determination that such content is to be displayed, and the dimming may be removed when the content is no longer displayed. In some instances, the dimming may be performed to mitigate artifacts caused by the pixel structure that enables dimming over the entire field of view.
At step 414, a plurality of spatially-resolved dimming values for the portion of the system field of view are determined based on the detected information. In some embodiments, the dimming values are determined using a formulaic approach based on a desired opacity or visibility of the virtual content. In one particular implementation, the visibility of the virtual content may be calculated using the following equation:
where Vis the visibility, Imax is the brightness of the virtual image light as indicated by the image information, Iback is related to a light value associated with the world object as indicated by the light information (which may be modified by the determined dimming value), and C is a desired contrast (e.g., 100:1). For example, the visibility equation may be used at each pixel location of the dimmer to calculate a dimming value for the particular pixel location using the brightness of the virtual image light at the particular pixel location and the light value associated with the world object at the particular pixel location. In some embodiments, Iback may be defined using the following equation:
Iback=Tv*Iworld
where Tv is the percentage of light that is allowed to pass through one or more pixels of the dimmer, and Iworld is the brightness of ambient light from the world as indicated by the light information. In some examples, Tv may be representative of or related to a dimming value.
At step 416, the dimmer is adjusted to reduce an intensity of the light associated with the object in the portion of the system field of view. For example, the dimmer may be adjusted such that the intensity of the light associated with the object impinging on each pixel location of the dimmer is reduced according to the dimming value determined for that particular pixel location. As used in the present disclosure, adjusting the dimmer may include initializing the dimmer, activating the dimmer, powering on the dimmer, modifying or changing a previously initialized, activated, and/or powered on dimmer, and the like. In some embodiments, the processor may send data to the dimmer indicating both the portion of the system field of view and the plurality of spatially-resolved dimming values.
At step 418, the projector is adjusted to adjust a brightness associated with the virtual image light. For example, in some embodiments it may be difficult to achieve a desired opacity or visibility of the virtual content without increasing or decreasing the brightness of the virtual object. In such embodiments, the brightness of the virtual image light may be adjusted before, after, simultaneously, or concurrently with adjusting the dimmer.
One of several inherent problems with using the pupil position to determine the gaze vector is illustrated in
Cone and rod locations may be estimated using information gathered by the eye tracker or, in some embodiments, the center position of the retinal layer corresponding to a high density of cones may be defined by continuing the gaze vector determined using the pupil position through the eye toward the back of the eye such that the gaze vector determined using the pupil position is co-linear with the gaze vector determined using cone and rod locations in high light conditions. In some embodiments, the AR device is configured such that the gaze vector is determined using cone and rod locations in low light conditions (e.g., “low light mode”) and is determined using the center of rotation of the eye in high light conditions. In such embodiments, a light threshold may be established that a detected light value may be evaluated against, causing the gaze vector to be determined using cone and rod locations when the detected light value is below the light threshold and causing the gaze vector to be determined using the center of rotation of the eye when the detected light value is above the light threshold.
In some embodiments in which the dimmed area is significantly large and/or the dimming values are significantly high, the detected ambient light using the light sensor of the AR device may not be indicative of the actual amount of light reaching the eye. In such embodiments, the size of the pupil may be used as a proxy for the amount of light reaching the eye. For example, the AR device may switch to a “low light mode” (causing the gaze vector to be determined using cone and rod locations) when the pupil size exceeds a pupil size threshold. For example, in some implementations the pupil size threshold may be set to be 20% above an average pupil size of a user in high light conditions (e.g., pupil size may correspond to area, diameter, circumference, etc. of the pupil). In another particular embodiment, the pupil size threshold may be predetermined based on average known pupil sizes in low light and high light conditions. Other possibilities are contemplated.
To dim in alignment with the perceived virtual content, vectors 1804 may be defined as intersecting each of points 1802 and the pupil position (i.e., a reference point). Intersection points 1806 may then be defined at each location where vectors 1804 intersect with the dynamic dimmer. As shown in reference to
A dynamic dimming element 2710 in this embodiment is mounted on the outside of the integrated stack. This allows switching from a transparent display for an AR mode to an opaque display completely blocking out the world light for a VR mode. The dimming element 2710 may correspond to a global dimming element or a pixelated dimming element. An external lens 2712 is positioned separate from the optical stack so as to provide a protective and/or supportive structure for the OST-HMD. External lens 2712 may also provide an amount of dimming to the entire system field of view.
In some embodiments, one or more components of local module 2910 may be similar to one or more components described with reference to
In some embodiments, display headset processor 2942 may receive virtual content data and pixelated dimmer data from a graphics processing unit (GPU) 2946 of remote module 2912 and may perform various correction and warping techniques prior to controlling pixelated dimmers 2903 and spatial light modulators 2904. Dimmer data generated by display headset processor 2942 may pass through one or more drivers which may modify or generate voltages for controlling dimmers 2903. In some embodiments, display headset processor 2942 may receive a depth image and a headset pose from sensor headset processor 2940 which can be used to improve the accuracy of the dimming and the projected virtual content.
Remote module 2912 may be electrically coupled to local module 2910 through one or more wired or wireless connections, and may be fixedly attached to the user or carried by the user, among other possibilities. Remote module 2912 may include a perception processing unit 2944 for performing/generating an environment lighting map, a headset pose, and eye sensing. Perception processing unit 2944 may send data to a CPU 2948 which may be configured to perform/generate passable world geometry and app scene geometry. CPU 2948 may send data to GPU 2946 which may be configured to perform a check for a minimum world luminance throughput, a dimming pixel alignment, a late frame time warp, and a render pipeline, among other operations. In some embodiments, CPU 2948 may be integrated with GPU 2946 such that a single processing unit may perform one or more of the functions described in reference to each. In some embodiments, the functionality of one or more of the components included in remote module 2912 may be similar to that of one or more components included in processing module 350.
At step 3002, a pixelated mask is generated. When method 3000 is used in conjunction with method 400, the pixelated mask may be generated based on the portion of the system field of view to be dimmed determined in step 412 and/or the dimming values determined in step 414 (i.e., dimmed area 236). In some embodiments, the pixelated mask may be generated using one or both of the techniques described in reference to
At step 3004, the dimmer is adjusted in accordance with the pixelated mask. For example, each pixel of the dimmer may be set to a particular dimming value as indicated by the pixelated mask. For examples in which method 3000 is used in conjunction with method 400, one or more operations of step 3004 may at least in part correspond to one or more operations of step 416.
At step 3006, the user looks through the dimmer and the observable dimming (to the user) is equivalent to the pixelated mask convolved with the PSF of a single pixel. Accordingly, step 3006 is inherently performed by the optics of the eye when the user is wearing the AR device, rather than being performed directly by a component of the AR device. On the other hand, steps 3002 and 3004 may, for example, be performed by one or more components of the AR device.
At step 3212, the PSF of a single pixel is estimated based on the pupil diameter, the accommodative state of the eye lens, the size/shape of a pixel, and the distance from a pixel to the pupil. Where the shape of a pixel can be approximated using a circle, the size/shape of a pixel may be represented as diameter h, the pupil diameter as diameter p, and the distance from a pixel to the pupil as distance d (using the nomenclature established in
At step 3214, a desired pixelated mask at the depth of the virtual content is generated. At step 3216, the desired pixelated mask is deconvolved using the estimated PSF of a single pixel. In some embodiments, the deconvolution is performed in the spatial frequency domain by dividing the Fourier Transform of the desired pixelated mask at the depth of the virtual content by the Fourier Transform of the estimated PSF of a single pixel and performing an inverse-Fourier Transform. Alternatively or additionally, the deconvolution may be performed dividing the Fourier Transform of the estimated PSF of a single pixel by the Fourier Transform of the desired pixelated mask at the depth of the virtual content and performing an inverse-Fourier Transform. The pixelated mask used subsequently in step 3204 is set to the result of the deconvolution.
As a result of performing method 3200, the observable dimming to the user (shown adjacent to step 3206) is significantly less blurred in comparison to the technique in method 3100 despite the dissimilarity between the pixelated mask (shown adjacent to step 3204) and the desired pixelated mask (shown adjacent to step 3214). In some embodiments, one or more steps of method 3200 may correspond to one or more steps of methods 3000 and 3100. In some embodiments, one or more steps of method 3200 may be omitted or modified.
Computer system 3300 is shown comprising hardware elements that can be electrically coupled via a bus 3305, or may otherwise be in communication, as appropriate. The hardware elements may include one or more processors 3310, including without limitation one or more general-purpose processors and/or one or more special-purpose processors such as digital signal processing chips, graphics acceleration processors, and/or the like; one or more input devices 3315, which can include without limitation a mouse, a keyboard, a camera, and/or the like; and one or more output devices 3320, which can include without limitation a display device, a printer, and/or the like.
Computer system 3300 may further include and/or be in communication with one or more non-transitory storage devices 3325, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as a random access memory (“RAM”), and/or a read-only memory (“ROM”), which can be programmable, flash-updateable, and/or the like. Such storage devices may be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and/or the like.
Computer system 3300 might also include a communications subsystem 3319, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and/or a chipset such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc., and/or the like. The communications subsystem 3319 may include one or more input and/or output communication interfaces to permit data to be exchanged with a network such as the network described below to name one example, other computer systems, television, and/or any other devices described herein. Depending on the desired functionality and/or other implementation concerns, a portable electronic device or similar device may communicate image and/or other information via the communications subsystem 3319. In other embodiments, a portable electronic device, e.g. the first electronic device, may be incorporated into computer system 3300, e.g., an electronic device as an input device 3315. In some embodiments, computer system 3300 will further comprise a working memory 3335, which can include a RAM or ROM device, as described above.
Computer system 3300 also can include software elements, shown as being currently located within the working memory 3335, including an operating system 3340, device drivers, executable libraries, and/or other code, such as one or more application programs 3345, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above, might be implemented as code and/or instructions executable by a computer and/or a processor within a computer; in an aspect, then, such code and/or instructions can be used to configure and/or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.
A set of these instructions and/or code may be stored on a non-transitory computer-readable storage medium, such as the storage device(s) 3325 described above. In some cases, the storage medium might be incorporated within a computer system, such as computer system 3300. In other embodiments, the storage medium might be separate from a computer system e.g., a removable medium, such as a compact disc, and/or provided in an installation package, such that the storage medium can be used to program, configure, and/or adapt a general purpose computer with the instructions/code stored thereon. These instructions might take the form of executable code, which is executable by computer system 3300 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on computer system 3300 e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc., then takes the form of executable code.
It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware might also be used, and/or particular elements might be implemented in hardware, software including portable software, such as applets, etc., or both. Further, connection to other computing devices such as network input/output devices may be employed.
As mentioned above, in one aspect, some embodiments may employ a computer system such as computer system 3300 to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by computer system 3300 in response to processor 3310 executing one or more sequences of one or more instructions, which might be incorporated into the operating system 3340 and/or other code, such as an application program 3345, contained in the working memory 3335. Such instructions may be read into the working memory 3335 from another computer-readable medium, such as one or more of the storage device(s) 3325. Merely by way of example, execution of the sequences of instructions contained in the working memory 3335 might cause the processor(s) 3310 to perform one or more procedures of the methods described herein. Additionally or alternatively, portions of the methods described herein may be executed through specialized hardware.
The terms “machine-readable medium” and “computer-readable medium,” as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In embodiments implemented using computer system 3300, various computer-readable media might be involved in providing instructions/code to processor(s) 3310 for execution and/or might be used to store and/or carry such instructions/code. In many implementations, a computer-readable medium is a physical and/or tangible storage medium. Such a medium may take the form of a non-volatile media or volatile media. Non-volatile media include, for example, optical and/or magnetic disks, such as the storage device(s) 3325. Volatile media include, without limitation, dynamic memory, such as the working memory 3335.
Common forms of physical and/or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and/or code.
Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor(s) 3310 for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and/or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and/or executed by computer system 3300.
The communications subsystem 3319 and/or components thereof generally will receive signals, and the bus 3305 then might carry the signals and/or the data, instructions, etc. carried by the signals to the working memory 3335, from which the processor(s) 3310 retrieves and executes the instructions. The instructions received by the working memory 3335 may optionally be stored on a non-transitory storage device 3325 either before or after execution by the processor(s) 3310.
The methods, systems, and devices discussed above are examples. Various configurations may omit, substitute, or add various procedures or components as appropriate. For instance, in alternative configurations, the methods may be performed in an order different from that described, and/or various stages may be added, omitted, and/or combined. Also, features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology evolves and, thus, many of the elements are examples and do not limit the scope of the disclosure or claims.
Specific details are given in the description to provide a thorough understanding of exemplary configurations including implementations. However, configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the configurations. This description provides example configurations only, and does not limit the scope, applicability, or configurations of the claims. Rather, the preceding description of the configurations will provide those skilled in the art with an enabling description for implementing described techniques. Various changes may be made in the function and arrangement of elements without departing from the spirit or scope of the disclosure.
Also, configurations may be described as a process which is depicted as a schematic flowchart or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Furthermore, examples of the methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a non-transitory computer-readable medium such as a storage medium. Processors may perform the described tasks.
Having described several example configurations, various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the disclosure. For example, the above elements may be components of a larger system, wherein other rules may take precedence over or otherwise modify the application of the technology. Also, a number of steps may be undertaken before, during, or after the above elements are considered. Accordingly, the above description does not bind the scope of the claims.
As used herein and in the appended claims, the singular forms “a”, “an”, and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, reference to “a user” includes a plurality of such users, and reference to “the processor” includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.
Also, the words “comprise”, “comprising”, “contains”, “containing”, “include”, “including”, and “includes”, when used in this specification and in the following claims, are intended to specify the presence of stated features, integers, components, or steps, but they do not preclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.
It is also understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and scope of the appended claims.
This application is a continuation of U.S. patent application Ser. No. 16/557,706, filed Aug. 30, 2019, now U.S. Pat. No. 11,170,565, issued Nov. 9, 2021, entitled “SPATIALLY-RESOLVED DYNAMIC DIMMING FOR AUGMENTED REALITY DEVICE,” which is a non-provisional of and claims the benefit of and priority to U.S. Provisional Patent Application No. 62/858,252, filed Jun. 6, 2019, entitled “SPATIALLY-RESOLVED DYNAMIC DIMMING FOR AUGMENTED REALITY DEVICE,” and U.S. Provisional Patent Application No. 62/725,993, filed Aug. 31, 2018, entitled “SPATIALLY-RESOLVED DYNAMIC DIMMING FOR AUGMENTED REALITY DEVICE,” which are hereby incorporated by reference in their entirety for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3632865 | Haskell et al. | Jan 1972 | A |
| 4204742 | Johnson et al. | May 1980 | A |
| 4984179 | Waldern | Jan 1991 | A |
| 5138555 | Albrecht | Aug 1992 | A |
| 5271093 | Hata et al. | Dec 1993 | A |
| 5311879 | Yamada et al. | May 1994 | A |
| 5311897 | Greer | May 1994 | A |
| 5422653 | Maguire, Jr. | Jun 1995 | A |
| 5526042 | Ozawa et al. | Jun 1996 | A |
| 5742264 | Inagaki et al. | Apr 1998 | A |
| 5933125 | Fernie et al. | Aug 1999 | A |
| 6011526 | Toyoshima et al. | Jan 2000 | A |
| 6151179 | Poss | Nov 2000 | A |
| 6441811 | Sawada et al. | Aug 2002 | B1 |
| 6917370 | Benton | Jul 2005 | B2 |
| 6947219 | Ou | Sep 2005 | B1 |
| 7023536 | Zhang et al. | Apr 2006 | B2 |
| 7038846 | Mandella | May 2006 | B2 |
| 7088440 | Buermann et al. | Aug 2006 | B2 |
| 7110100 | Buermann et al. | Sep 2006 | B2 |
| 7113270 | Buermann et al. | Sep 2006 | B2 |
| 7161664 | Buermann et al. | Jan 2007 | B2 |
| 7203384 | Carl | Apr 2007 | B2 |
| 7268956 | Mandella | Sep 2007 | B2 |
| 7386799 | Clanton et al. | Jun 2008 | B1 |
| 7474809 | Carl et al. | Jan 2009 | B2 |
| 7729515 | Mandella et al. | Jun 2010 | B2 |
| 7758185 | Lewis | Jul 2010 | B2 |
| 7826641 | Mandella et al. | Nov 2010 | B2 |
| 7961909 | Mandella et al. | Jun 2011 | B2 |
| 8223024 | Petrou | Jul 2012 | B1 |
| 8508830 | Wang | Aug 2013 | B1 |
| 8542219 | Carl et al. | Sep 2013 | B2 |
| 8553935 | Mandella et al. | Oct 2013 | B2 |
| 8696113 | Lewis | Apr 2014 | B2 |
| 8730129 | Solomon | May 2014 | B2 |
| 8757812 | Melville et al. | Jun 2014 | B2 |
| 8831278 | Fedorovskaya et al. | Sep 2014 | B2 |
| 8832233 | Brin et al. | Sep 2014 | B1 |
| 8879155 | Teller | Nov 2014 | B1 |
| 8897494 | Mandella et al. | Nov 2014 | B2 |
| 8970709 | Gonzalez-banos et al. | Mar 2015 | B2 |
| 9189856 | Gonzalez-banos et al. | Nov 2015 | B1 |
| 9229540 | Mandella et al. | Jan 2016 | B2 |
| 9235934 | Mandella et al. | Jan 2016 | B2 |
| 9239473 | Lewis | Jan 2016 | B2 |
| 10185147 | Lewis | Jan 2019 | B2 |
| 10670808 | Trail | Jun 2020 | B1 |
| 11170565 | Mathur et al. | Nov 2021 | B2 |
| 20010035870 | Takeuchi et al. | Nov 2001 | A1 |
| 20030014212 | Ralston et al. | Jan 2003 | A1 |
| 20030137449 | Vashisth et al. | Jul 2003 | A1 |
| 20030156253 | Watanabe et al. | Aug 2003 | A1 |
| 20030158654 | Morita | Aug 2003 | A1 |
| 20040119715 | Everett et al. | Jun 2004 | A1 |
| 20040130520 | Maeda et al. | Jul 2004 | A1 |
| 20040178894 | Janssen | Sep 2004 | A1 |
| 20040258314 | Hashimoto | Dec 2004 | A1 |
| 20050046953 | Repetto et al. | Mar 2005 | A1 |
| 20050110732 | Kim | May 2005 | A1 |
| 20050148388 | Vayra et al. | Jul 2005 | A1 |
| 20050248852 | Yamasaki | Nov 2005 | A1 |
| 20050253055 | Sprague et al. | Nov 2005 | A1 |
| 20050254135 | Ou | Nov 2005 | A1 |
| 20060007056 | Ou | Jan 2006 | A1 |
| 20060105838 | Mullen | May 2006 | A1 |
| 20060109280 | Dawson | May 2006 | A1 |
| 20060132915 | Yang et al. | Jun 2006 | A1 |
| 20060214911 | Miller | Sep 2006 | A1 |
| 20060226231 | Johnston et al. | Oct 2006 | A1 |
| 20060227151 | Bannai | Oct 2006 | A1 |
| 20060256110 | Okuno et al. | Nov 2006 | A1 |
| 20060267889 | Kimura | Nov 2006 | A1 |
| 20060284791 | Chen et al. | Dec 2006 | A1 |
| 20070086668 | Ackley et al. | Apr 2007 | A1 |
| 20080005702 | Skourup et al. | Jan 2008 | A1 |
| 20080058629 | Seibel et al. | Mar 2008 | A1 |
| 20080059578 | Albertson et al. | Mar 2008 | A1 |
| 20080062131 | Chan et al. | Mar 2008 | A1 |
| 20080071559 | Arrasvuori | Mar 2008 | A1 |
| 20080186255 | Cohen et al. | Aug 2008 | A1 |
| 20080199049 | Daly | Aug 2008 | A1 |
| 20080215974 | Harrison et al. | Sep 2008 | A1 |
| 20080215975 | Harrison et al. | Sep 2008 | A1 |
| 20080215994 | Harrison et al. | Sep 2008 | A1 |
| 20080276178 | Fadell et al. | Nov 2008 | A1 |
| 20080278480 | Katano | Nov 2008 | A1 |
| 20090003662 | Joseph et al. | Jan 2009 | A1 |
| 20090066690 | Harrison | Mar 2009 | A1 |
| 20090089685 | Mordecai et al. | Apr 2009 | A1 |
| 20090147331 | Ashkenazi | Jun 2009 | A1 |
| 20090164916 | Jeong et al. | Jun 2009 | A1 |
| 20090177042 | Johnston | Jul 2009 | A1 |
| 20090187389 | Dobbins et al. | Jul 2009 | A1 |
| 20090213114 | Dobbins et al. | Aug 2009 | A1 |
| 20090222424 | Van | Sep 2009 | A1 |
| 20090225001 | Biocca et al. | Sep 2009 | A1 |
| 20090285484 | Mallinson et al. | Nov 2009 | A1 |
| 20090293012 | Alter et al. | Nov 2009 | A1 |
| 20090316116 | Melville et al. | Dec 2009 | A1 |
| 20090322671 | Scott et al. | Dec 2009 | A1 |
| 20100020216 | Christian et al. | Jan 2010 | A1 |
| 20100060647 | Brown et al. | Mar 2010 | A1 |
| 20100085462 | Sako et al. | Apr 2010 | A1 |
| 20100110368 | Chaum | May 2010 | A1 |
| 20100137684 | Shibasaki et al. | Jun 2010 | A1 |
| 20100141555 | Rorberg et al. | Jun 2010 | A1 |
| 20100149073 | Chaum et al. | Jun 2010 | A1 |
| 20100197390 | Craig et al. | Aug 2010 | A1 |
| 20100260426 | Huang et al. | Oct 2010 | A1 |
| 20100287500 | Whitlow et al. | Nov 2010 | A1 |
| 20110075902 | Song et al. | Mar 2011 | A1 |
| 20110096337 | Hirose et al. | Apr 2011 | A1 |
| 20110227812 | Haddick et al. | Sep 2011 | A1 |
| 20110234879 | Kashitani | Sep 2011 | A1 |
| 20110246908 | Akram et al. | Oct 2011 | A1 |
| 20110267361 | Kurozuka | Nov 2011 | A1 |
| 20110313653 | Lindner | Dec 2011 | A1 |
| 20120050140 | Border et al. | Mar 2012 | A1 |
| 20120068913 | Bar-zeev et al. | Mar 2012 | A1 |
| 20120086728 | Mcardle et al. | Apr 2012 | A1 |
| 20120087580 | Woo et al. | Apr 2012 | A1 |
| 20120089949 | Chen et al. | Apr 2012 | A1 |
| 20120092328 | Flaks et al. | Apr 2012 | A1 |
| 20120113092 | Bar-zeev et al. | May 2012 | A1 |
| 20120120103 | Border et al. | May 2012 | A1 |
| 20120127062 | Bar-zeev et al. | May 2012 | A1 |
| 20120127136 | Schneider et al. | May 2012 | A1 |
| 20120127284 | Bar-zeev et al. | May 2012 | A1 |
| 20120154277 | Bar-zeev et al. | Jun 2012 | A1 |
| 20120154557 | Perez et al. | Jun 2012 | A1 |
| 20120169752 | Kurozuka | Jul 2012 | A1 |
| 20120182313 | Ahn et al. | Jul 2012 | A1 |
| 20120183137 | Laughlin | Jul 2012 | A1 |
| 20120188148 | Dejong | Jul 2012 | A1 |
| 20120194554 | Kaino et al. | Aug 2012 | A1 |
| 20120200600 | Demaine | Aug 2012 | A1 |
| 20120206452 | Geisner et al. | Aug 2012 | A1 |
| 20120212399 | Border et al. | Aug 2012 | A1 |
| 20120218301 | Miller | Aug 2012 | A1 |
| 20120236262 | Johansson | Sep 2012 | A1 |
| 20120242560 | Nakada et al. | Sep 2012 | A1 |
| 20120244939 | Braun | Sep 2012 | A1 |
| 20120249586 | Wither et al. | Oct 2012 | A1 |
| 20120249797 | Haddick et al. | Oct 2012 | A1 |
| 20120327116 | Liu et al. | Dec 2012 | A1 |
| 20130021226 | Bell | Jan 2013 | A1 |
| 20130021373 | Vaught et al. | Jan 2013 | A1 |
| 20130042296 | Hastings et al. | Feb 2013 | A1 |
| 20130044128 | Liu et al. | Feb 2013 | A1 |
| 20130050186 | Large et al. | Feb 2013 | A1 |
| 20130050258 | Liu et al. | Feb 2013 | A1 |
| 20130050260 | Reitan | Feb 2013 | A1 |
| 20130050432 | Perez et al. | Feb 2013 | A1 |
| 20130050642 | Lewis et al. | Feb 2013 | A1 |
| 20130083011 | Geisner et al. | Apr 2013 | A1 |
| 20130083173 | Geisner et al. | Apr 2013 | A1 |
| 20130093788 | Liu et al. | Apr 2013 | A1 |
| 20130114043 | Balan et al. | May 2013 | A1 |
| 20130117377 | Miller | May 2013 | A1 |
| 20130120224 | Cajigas et al. | May 2013 | A1 |
| 20130120706 | Kakinuma et al. | May 2013 | A1 |
| 20130141419 | Mount et al. | Jun 2013 | A1 |
| 20130141434 | Sugden | Jun 2013 | A1 |
| 20130156266 | Horii | Jun 2013 | A1 |
| 20130194164 | Sugden et al. | Aug 2013 | A1 |
| 20130208014 | Fleck et al. | Aug 2013 | A1 |
| 20130215230 | Miesnieks et al. | Aug 2013 | A1 |
| 20130249946 | Kimura | Sep 2013 | A1 |
| 20130257748 | Ambrus et al. | Oct 2013 | A1 |
| 20130286004 | Mcculloch et al. | Oct 2013 | A1 |
| 20130293468 | Perez et al. | Nov 2013 | A1 |
| 20130307875 | Anderson | Nov 2013 | A1 |
| 20130342564 | Kinnebrew et al. | Dec 2013 | A1 |
| 20130342569 | Karkkainen et al. | Dec 2013 | A1 |
| 20130342571 | Kinnebrew et al. | Dec 2013 | A1 |
| 20140002496 | Lamb et al. | Jan 2014 | A1 |
| 20140064557 | Hara et al. | Mar 2014 | A1 |
| 20140071163 | Kinnebrew et al. | Mar 2014 | A1 |
| 20140098137 | Fein et al. | Apr 2014 | A1 |
| 20140098425 | Schon et al. | Apr 2014 | A1 |
| 20140118829 | Ma et al. | May 2014 | A1 |
| 20140132484 | Pandey et al. | May 2014 | A1 |
| 20140139551 | Mcculloch et al. | May 2014 | A1 |
| 20140145997 | Tiruvuru | May 2014 | A1 |
| 20140168035 | Luebke et al. | Jun 2014 | A1 |
| 20140168783 | Luebke et al. | Jun 2014 | A1 |
| 20140178029 | Raheman et al. | Jun 2014 | A1 |
| 20140192085 | Kim | Jul 2014 | A1 |
| 20140204077 | Kamuda et al. | Jul 2014 | A1 |
| 20140218361 | Abe et al. | Aug 2014 | A1 |
| 20140222432 | Ahn et al. | Aug 2014 | A1 |
| 20140267402 | Hing et al. | Sep 2014 | A1 |
| 20140354685 | Lazarow et al. | Dec 2014 | A1 |
| 20140368532 | Keane et al. | Dec 2014 | A1 |
| 20150097865 | Subramanian et al. | Apr 2015 | A1 |
| 20150205126 | Schowengerdt | Jul 2015 | A1 |
| 20150248046 | Schowengerdt | Sep 2015 | A1 |
| 20150254793 | Hastings et al. | Sep 2015 | A1 |
| 20150301356 | Tabirian et al. | Oct 2015 | A1 |
| 20160047956 | Tabirian et al. | Feb 2016 | A1 |
| 20160055822 | Bell | Feb 2016 | A1 |
| 20160063706 | Gonzalez-banos et al. | Mar 2016 | A1 |
| 20160093106 | Black | Mar 2016 | A1 |
| 20160179336 | Ambrus et al. | Jun 2016 | A1 |
| 20160327798 | Xiao | Nov 2016 | A1 |
| 20170010469 | Samec et al. | Jan 2017 | A1 |
| 20170168307 | Itani | Jun 2017 | A1 |
| 20170223344 | Kaehler | Aug 2017 | A1 |
| 20180053284 | Rodriguez et al. | Feb 2018 | A1 |
| 20180129048 | Robbins et al. | May 2018 | A1 |
| 20180180890 | Baerenrodt et al. | Jun 2018 | A1 |
| 20180188528 | Browy et al. | Jul 2018 | A1 |
| 20180188536 | Bell et al. | Jul 2018 | A1 |
| 20180314066 | Bell | Nov 2018 | A1 |
| 20180356639 | Schaefer et al. | Dec 2018 | A1 |
| 20190088021 | Tanaka et al. | Mar 2019 | A1 |
| 20200058256 | Seibert | Feb 2020 | A1 |
| 20200074724 | Mathur et al. | Mar 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 112639579 | Apr 2021 | CN |
| 2649485 | Oct 2013 | EP |
| 3844559 | Jul 2021 | EP |
| 3914959 | Dec 2021 | EP |
| 6010224 | Jan 1985 | JP |
| 2006293604 | Oct 2006 | JP |
| 2011175439 | Sep 2011 | JP |
| 2012505430 | Mar 2012 | JP |
| 2014505899 | Mar 2014 | JP |
| 2012020527 | Feb 2012 | WO |
| 2012078410 | Jun 2012 | WO |
| 2016181108 | Nov 2016 | WO |
| 2020047486 | Mar 2020 | WO |
| 2020146760 | Jul 2020 | WO |
| 2021002641 | Jan 2021 | WO |
| Entry |
|---|
| Application No. EP19854491.8, Extended European Search Report, dated Sep. 24, 2021, 8 pages. |
| U.S. Appl. No. 16/158,041, Non-Final Office Action dated Mar. 8, 2021, 10 pages. |
| U.S. Appl. No. 16/158,041, Non-Final Office Action dated Apr. 28, 2020, 9 pages. |
| U.S. Appl. No. 16/557,706, Advisory Action dated Sep. 10, 2020, 3 pages. |
| U.S. Appl. No. 16/557,706, Final Office Action dated Jul. 2, 2020, 31 pages. |
| U.S. Appl. No. 16/557,706, Non-Final Office Action dated Apr. 15, 2020, 27 pages. |
| U.S. Appl. No. 16/557,706, Non-Final Office Action dated Oct. 14, 2020, 30 pages. |
| U.S. Appl. No. 16/557,706, Notice of Allowance dated Feb. 24, 2021, 7 pages. |
| Aronoff et al., Collaborative Augmented Reality for Better Standards, Complex Systems Concurrent Engineering, 2007, 8 pages. |
| Azuma, A Survey of Augmented Reality, In Presence: Teleoperators and Virtual Environment, vol. 6, Aug. 1997, pp. 355-385. |
| Bimber et al., Spatial Augmented Reality, Merging Real and Virtual Worlds, 2005, 393 pages. |
| Chen et al., Electrically Adjustable Location of a Projected Image in Augmented Reality via a Liquid-Crystal Lens, Optics Express, vol. 23, No. 22, Nov. 2015, 9 pages. |
| European Application No. 18818480.8, Extended European Search Report dated Feb. 23, 2021, 7 pages. |
| Freeman et al., Scanned Laser Pico-Projectors: Seeing the Big Picture with a Small Device, Optics and Photonics News, vol. 20, Issue 5, May 2009, pp. 28-34. |
| International Application No. PCT/US2019/049176, International Preliminary Report on Patentability dated Mar. 11, 2021, 7 pages. |
| International Application No. PCT/US2019/049176, International Search Report and Written Opinion dated Nov. 14, 2019, 8 pages. |
| International Application No. PCT/US2020/013130, International Search Report and Written Opinion dated May 22, 2020, 15 pages. |
| Roscher et al., Low-Cost Projection Device with a 2D Resonant Microscanning Mirror, MOEMS Display and Imaging Systems II, vol. 5348, Jan. 24, 2004, pp. 22-31. |
| Application No. AU2018284089, “First Examination Report”, dated Apr. 21, 2022, 3 pages. |
| Application No. EP20738194.8, Extended European Search Report, dated Feb. 7, 2022, 10 pages. |
| Application No. IN201947053419, “Examination Report”, dated Feb. 25, 2022, 6 pages. |
| Application No. IN202047018987, “Examination Report”, dated Mar. 29, 2022, 5 pages. |
| Application No. JP2019-568041, Office Action, dated Apr. 27, 2022, 6 pages [no translation available]. |
| Application No. PCT/US2020/013130, International Preliminary Report on Patentability, dated Jul. 22, 2021, 12 pages. |
| Application No. JP 2020519432, “Office Action”, dated Jul. 11, 2022, 2 pages [no English Translation available]. |
| Number | Date | Country | |
|---|---|---|---|
| 20210350614 A1 | Nov 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62858252 | Jun 2019 | US | |
| 62725993 | Aug 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16557706 | Aug 2019 | US |
| Child | 17330146 | US |
