Patent Grant
11249310
The examples relate generally to augmented reality devices, and in particular to an augmented reality device with an external light control layer for realtime contrast control.
In an optical see-through augmented reality (AR) head-wearable device, it is desirable to have an appropriate balance of external light from a real-world scene versus internal light associated with the AR imagery. For example, if the external light is substantially brighter than the internal light, then the AR imagery may appear washed out or may not even be visible. If the internal light is much brighter than the external light, then the real-world scene may be difficult to see, and excess light may escape from the head-wearable device, allowing the user to be seen by other individuals in situations where it may be desirable to keep light emissions low, such as in a tactical situation.
The embodiments disclosed herein relate to an augmented reality (AR) device with an external light control layer, such as a photochromic or electrophotochromic layer, for realtime contrast control. The embodiments ensure that computer-generated AR imagery is easily visible to a user even as the user transitions from dark to bright areas of the real world, such as when moving from an outdoor scene on a bright day to an interior of a dark building, and ensure that the internal light associated with computer-generated AR imagery is not overly bright with respect to the external light from the scene.
In one embodiment a device is provided. The device includes an integrated lens system having a field of view (FOV). The integrated lens system includes a reflective layer, a transparent substrate layer, an external light control layer, and an anti-reflective layer. The device also includes a display device configured to present an image on an interior surface of the integrated lens system.
In another embodiment another device is provided. The device includes an integrated lens system having an FOV. The integrated lens system includes a reflective layer, a transparent substrate layer, an electrophotochromic layer, and an anti-reflective layer. The device also includes a display device configured to present an image on an interior surface of the integrated lens system. The device further includes a light sensor configured to generate an external light power level that quantifies an amount of light received from a scene within the FOV. The device further includes a controller communicatively coupled to the electrophotochromic layer and the display device.
Those skilled in the art will appreciate the scope of the disclosure and realize additional aspects thereof after reading the following detailed description of the embodiments in association with the accompanying drawing figures.
The accompanying drawing figures incorporated in and forming a part of this specification illustrate several aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
The embodiments set forth below represent the information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure and the accompanying claims.
Any flowcharts discussed herein are necessarily discussed in some sequence for purposes of illustration, but unless otherwise explicitly indicated, the embodiments are not limited to any particular sequence of steps. The use herein of ordinals in conjunction with an element is solely for distinguishing what might otherwise be similar or identical labels, such as “first message” and “second message,” and does not imply a priority, a type, an importance, or other attribute, unless otherwise stated herein. The term “about” used herein in conjunction with a numeric value means any value that is within a range of ten percent greater than or ten percent less than the numeric value. As used herein and in the claims, the articles “a” and “an” in reference to an element refers to “one or more” of the element unless otherwise explicitly specified.
The embodiments disclosed herein relate to an augmented reality (AR) device with an external light control layer for realtime contrast control. In some embodiments the external light control layer comprises a photochromic layer. In other embodiments the external light control layer comprises an electrophotochromic layer. The embodiments ensure that computer-generated AR imagery is easily visible to a user even as the user transitions from dark to bright areas of the real world, such as when moving from an outdoor scene on a bright day to an interior of a dark building, and ensures that the internal light associated with computer-generated AR imagery is not overly bright with respect to the external light from the scene.
The AR device 1 may also include a frame structure (not illustrated in
The integrated lens system 2 includes a plurality of layers 6-1-6-7 (generally, layers 6), some of which are optional. A reflective layer 6-1 is configured to reflect images provided from the display device 4 to an eye of a user. The reflective layer 6-1 may comprise a coating on top of a transparent substrate layer 6-2, which comprises a see-through substrate of the integrated lens system 2. In some embodiments, the transparent substrate layer 6-2 may be a polycarbonate material, but the embodiments are not limited to any particular transparent substrate. In some embodiments, the transparent substrate layer 6-2 is a reflective optical surface made in accordance with mechanisms disclosed in U.S. Pat. No. 8,781,794 entitled “Methods and Systems for Creating Free Space Reflective Optical Surfaces” and U.S. Pat. No. 8,625,200 entitled “Head-Mounted Display Apparatus Employing One or More Reflective Optical Surfaces,” the contents of both of which are incorporated herein by reference.
The integrated lens system 2 also includes an external light control layer, which in this embodiment comprises a photochromic layer 6-3. The transmissivity of the photochromic layer 6-3 changes in response to an amount of external light from a scene 7 that impinges upon the integrated lens system 2. Examples of a photochromic layer 6-3 include, by way of non-limiting example, a transition lens that automatically darkens in response to impingement of light. While for purposes of illustration and explanation the layers 6 are discussed separately, in practice, some of the layers 6 may comprise a single physical layer. For example the photochromic layer 6-3 and the transparent substrate layer 6-2 may be integrated with one another, and in practice comprise a single physical layer that provides both a suitable transparent substrate and a photochromic function. Similarly, the transparent substrate layer 6-2 and the reflective layer 6-1 may in practice comprise a single physical layer that provides both a suitable transparent substrate and suitable reflective properties.
Although
While for purposes of illustration
In some embodiments, the integrated lens system 2 may also include an anti-scratch layer 6-5 that is bonded, coated, or otherwise attached to the anti-reflective layer 6-4 to protect the external surface of the integrated lens system 2. The integrated lens system 2 may also include a protective layer 6-6 that is bonded, coated, or otherwise attached to the reflective layer 6-1, and an anti-fog layer 6-7 that is coupled to the protective layer 6-6 to keep the interior surface of the integrated lens 2, which may be in relatively close proximity to the user's face, from fogging.
In operation, external light 8 from the scene 7 is transmitted through the integrated lens system 2 and presented to the eyes of the user. In response to the amount of external light 8, the photochromic layer 6-3 automatically darkens (or lightens) to alter the transmissivity of the photochromic layer 6-3, thereby facilitating a strong contrast between the scene 7 seen by the user and that of the AR imagery provided by the display device 4 on the internal surface of the integrated lens system 2.
Although
While for purposes of illustration
The AR device 10 includes a light power sensor 30 that is configured to generate an external light power level 32 that corresponds to an amount of external light 34 received from a scene within the FOV. The external light power level 32 may be an analog or digital signal. The external light power level 32 may be quantified in any desired manner. In one embodiment, the quantification of the amount of external light 34 may be based on irradiance measured in watts per square meter, or may be based on luminosity measured in lumens, (candela/meter)2, or in any other appropriate units.
The term “external light” in this context refers to light received from outside the AR device 10, in contrast to light generated by the AR device 10, such as internal light 36 generated by the display device 14. The internal light 36 is in the form of imagery provided to the display device 14 and presented by the display device 14. As will be discussed below in greater detail, in some embodiments the controller 18 may control the power level associated with such imagery. The display device 14 may comprise any suitable display device, such as a spatial light modulator, or the like.
The controller 18 may comprise any suitable control mechanism, including a processor device and memory, programmed to implement the functionality described herein. In some embodiments, the controller 18 may comprise a fuzzy logic controller that utilizes fuzzy logic to control various aspects of the AR device 10, as described in greater detail herein. Generally, the controller 18 utilizes one or more inputs to determine a control signal to send to the electrophotochromic layer 20 to alter the transmissivity of the electrophotochromic layer 20 in order to limit the amount of external light 34 that is presented to the eye 24 of the user. In one embodiment, the controller 18 utilizes the external light power level 32 to determine a control signal 38 that alters the transmissivity of the electrophotochromic layer 20. The controller 18 may generate the control signal 38 to ensure that the contrast between the external light 34 presented to the user and the light associated with an image generated by the display device 14 has a desired contrast ratio between the scene and the image. While solely for purposes of illustration the controller 18 is discussed herein as being a fuzzy logic controller, the embodiments are not limited to any particular type of controller.
The AR device 10 includes, or is communicatively coupled to, an AR computing device 40 that is responsible for generating AR imagery, and sending the AR imagery to the display device 14 for presentation on the interior surface 16 of the integrated lens system 12 for reflection into the eye 24 of the user. In some embodiments, the AR computing device 40 may determine an initial power level 42 associated with an image, and provide the initial power level 42 to the controller 18. The initial power level 42 may be quantified in any desired manner. In one embodiment, the initial power level 42 may be determined by the sum of the red, green, blue (RGB) values for each pixel image, multiplied by a constant related to the power of the display device 14, and multiplied by a ratio of the reflection size of the internal surface 16 to the size of the display device 14. Basically, any number of mechanisms may be used to determine the initial power level 42, so long as the formula is associated with how bright any particular image will be. In some embodiments, the determination may be based simply on a peak brightness of the brightest pixel in the image.
Based on the external light power level 32 of the external light 34, the controller 18 may determine a different power level 44 of the image, and provide the power level 44 to the AR computing device 40. The AR computing device 40 may then alter the image to meet the power level 44, such as by altering RGB values of the pixels of the image to increase or decrease the brightness of the image. The AR computing device 40 then sends the image to the display device 14.
In some embodiments, the electrophotochromic layer 20 comprises a voltage-controlled optical filter that has a control voltage that can be within a range between a lowest control voltage and a highest control voltage associated with the particular electrophotochromic layer 20. A lowest control voltage may correspond to a maximum transmissivity of the electrophotochromic layer 20, and a highest control voltage may correspond to a minimum transmissivity of the electrophotochromic layer 20. The controller 18 may attempt to maximize the contrast ratio of the internal light 36 to that of the external light 34 that is presented to the eye 24 of the user, while maintaining a level of brightness that is within the acceptable and optimum range for the eye 24. The contrast ratio may be defined as
where Imax is the power level of the internal light 36 and Imin is the external light power level 32 of the external light 34. In some embodiments, the AR device 10 includes a pupil measurement device 46 that is configured to determine a pupil size 47 of a pupil 48 of the eye 24. The AR device 10 may use the pupil size 47 of the pupil 48 to determine whether the combination of the internal light 36 and the external light 34 is within a desired “optimal” range. In such embodiments, the controller 18 may attempt to keep the internal light 36 and the external light 34 at a relatively constant level “Iopt” with the highest possible contrast ratio as the user wears the AR device 10 through different environmental lighting conditions. Thus, the AR device 10 may utilize measurements such as the external light power level 32 of the external light 34, the initial power level 42 of a next image to be presented on the display device 14 and the pupil size 47 to determine the control signal 38 and the power level 44 to both maintain a desired contrast ratio between the internal light 36 and the external light 34 and to maintain the total amount of the internal light 36 and the external light 34 within an “optimum” range.
It should be noted that these calculations, in operation, may be occurring continuously at a relatively rapid rate, such as 30, 60 or 120 frames per second. Thus, each 1/120th of a second the controller may determine the current external light power level 32 of the external light 34, determine the current pupil size 47 from the pupil measurement device 46, and may be provided an initial power level 42 of a next image to be presented by the AR computing device 40 on the display device 14. In response, the controller 18 determines the appropriate control signal 38 and sends the control signal 38 to the electrophotochromic layer 20, and also determines the appropriate power level 44 of the image, and sends the power level 44 to the AR computing device 40. The AR computing device 40 alters the image as necessary to meet the power level 44, and sends the image to the display device 14. Moreover, while for purposes of illustration only a single display device 14 is illustrated, in some embodiments the AR device 10 includes two display devices 14, one for each eye 24 of the user.
The light power sensor 30 receives external light 34 from a scene 56 within an FOV 58 of the lens system 12, and determines an external light power level 32 that quantifies the amount of external light 34 received from the scene 56. The scene 56 is transmitted through the integrated lens system 12, depending on the transmissivity of the electrophotochromic layer 20, to the eyes of the user. Note that the scene 56 continuously changes as the FOV 58 encompasses different scenes 56 in the real world as the user moves about the real world. Thus, the amount of external light 34 also continuously changes, and the light power sensor 30 continuously determines the external light power level 32 and provides the external light power level 32 to the controller 18. This may occur, for example, every 10 milliseconds, every 100 milliseconds, or at any other desired interval.
In some embodiments, the controller 18 comprises a fuzzy logic controller 18, and relies on fuzzy logic to determine the control signal 38. In one embodiment, wherein the AR device 10 utilizes only the external light power level 32 to generate the control signal 38, and does not alter the image power of the imagery presented on the display device 14, the fuzzy logic controls may be summarized in the following manner:
In this example, fuzzy membership rules 4 (bright) and 5 (veryBright) are activated because the external light power level 32 falls into the ranges associated with fuzzy membership functions 4 and 5. The fuzzy membership rules map input membership functions to output membership functions. The amount that each fuzzy input membership function is activated is shown in cross-hatching 74. The consequents for the fuzzy membership functions are shown in the five plots 76-1-76-5 on the right side, which, in this example, have sigma-shaped membership functions. It is seen that the membership functions for the fuzzy membership rules 4 and 5 are activated as indicated by cross-hatching 78, to levels prescribed by the membership illustrated in plots 72-4 and 72-5. A plot 80 is the sum of all plots 76-1-76-5. A centroid 82 of the area 84 of the plot 80 is shown as having a value of 0.784, which is then used as the control signal 38 to alter the transmissivity of the electrophotochromic layer 20.
Thus, as the value of the external light power level 32 changes, different fuzzy input membership functions 70-1-70-5 will activate, resulting in a smoothly changing control signal 38.
In another embodiment the fuzzy logic controller 18 uses the external light power level 32 and the initial power level 42 of the image next to be presented on the display device 14, alters the image power of the next image to be presented on the display device 14, and ensures that the contrast ratio of the external light 34 from the scene 56 presented to the eyes of the user 60 and the light of the imagery being depicted on the internal surface 16 of the integrated lens system 12 stays within a desired range. In this embodiment, the contrast ratio may be represented as the ratio of the external light power level 32 to the image power level 44, although other mechanisms may be used. In this embodiment, the fuzzy logic controller 18 determines two outputs, the control signal 38 to set the transmissivity of the electrophotochromic layer 20 and the power level 44 to set the power level of the next image to be presented on the display device 14.
The fuzzy logic rules may then be summarized as:
An external light power level 32 of 0.68, which is relatively high, and an initial power level 42 of 0.2, which is relatively low, results in an external control signal 38 of 0.3 and a power level 44 of 0.5. Thus, in this example, the transmissivity of the electrophotochromic layer 20 is lowered, and the power of the next image to be presented on the display device 14 is increased.
In other embodiments, the fuzzy logic controller 18 may utilize the pupil size 47, which may alter the determined external control signal 38 and power level 44. In one embodiment, the goal of the fuzzy logic controller 18 is to maintain the pupil size 47 at a nominal level. This aids in ensuring a safe light power level is provided to the eye.
In some embodiments, the fuzzy logic rules can be adjusted to be changed per user 60. Override fuzzy logic may be installed to maximize brightness, or for different operational modes. For example, if a user 60 is aware that they have important work to perform in a class that requires less outside light interference, the user can choose a mode that has different fuzzy logic rules.
Among other advantages, the AR device 10 dynamically maintains an optimized balance of external light to internal light under rapidly varying light conditions. The AR device 10 can thus be effectively used in a wide regime of operations. The rapid control and alteration of light transmission through the electrophotochromic layer 20, coupled with the feedback loop driven by the external light power sensor 30, and the internal light power level 44, facilitates optimizing the contrast ratio between the scene that is presented to the user 60 and the imagery that is concurrently presented to the user 60, to allow the AR imagery to be viewed reliably on a continuous basis in a wide array of external conditions.
The system bus 94 may be any of several types of bus structures that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and/or a local bus using any of a variety of commercially available bus architectures. The memory 92 may include non-volatile memory 95 (e.g., read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.), and volatile memory 96 (e.g., random-access memory (RAM)). A basic input/output system (BIOS) 98 may be stored in the non-volatile memory 95 and can include the basic routines that help to transfer information between elements within the AR device 10. The volatile memory 96 may also include a high-speed RAM, such as static RAM, for caching data.
The AR device 10 may further include or be coupled to a non-transitory computer-readable storage medium such as a storage device 100, which may comprise, for example, an internal or external hard disk drive (HDD) (e.g., enhanced integrated drive electronics (EIDE) or serial advanced technology attachment (SATA)), HDD (e.g., EIDE or SATA) for storage, flash memory, or the like. The storage device 100 and other drives associated with computer-readable media and computer-usable media may provide non-volatile storage of data, data structures, computer-executable instructions, and the like. Although the description of computer-readable media above refers to an HDD, it should be appreciated that other types of media that are readable by a computer, such as Zip disks, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the operating environment, and, further, that any such media may contain computer-executable instructions for performing novel methods of the disclosed examples. A number of modules can be stored in the storage device 100 and in the volatile memory 96, including an operating system and one or more program modules, such as the controller 18, which may implement the functionality described herein in whole or in part.
All or a portion of the examples may be implemented as a computer program product 102 stored on a transitory or non-transitory computer-usable or computer-readable storage medium, such as the storage device 100, which includes complex programming instructions, such as complex computer-readable program code, to cause the processor device 91 to carry out the steps described herein. Thus, the computer-readable program code can comprise software instructions for implementing the functionality of the examples described herein when executed on the processor device 91.
The AR device 10 may also include a communications interface 104, such as an Ethernet transceiver or the like, suitable for communicating with a network, or the AR computing device 40, as appropriate or desired. The AR device 10 includes the display device 14, the light power sensor 30, the optional pupil measurement device 46, and the integrated lens system 12.
Those skilled in the art will recognize improvements and modifications to the preferred embodiments of the disclosure. All such improvements and modifications are considered within the scope of the concepts disclosed herein and the claims that follow.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8625200 | Smith et al. | Jan 2014 | B2 |
| 8781794 | Harrison et al. | Jul 2014 | B2 |
| 20120001833 | Lindig | Jan 2012 | A1 |
| 20120249797 | Haddick | Oct 2012 | A1 |
| 20130057937 | Berman | Mar 2013 | A1 |
| 20130127980 | Haddick | May 2013 | A1 |
| 20160033772 | Han | Feb 2016 | A1 |
| 20170293356 | Khaderi | Oct 2017 | A1 |
| 20180314066 | Bell | Nov 2018 | A1 |
| 20190227347 | Jimenez | Jul 2019 | A1 |
