Patent Application
20250148642
This disclosure relates generally to Artificial Reality (AR) and Virtual Reality (VR), and in particular to calibration of AR/VR head mounted devices.
Eye tracking (ET) during an AR/VR session is becoming increasingly important for many applications and features for head mounted devices. Eye tracking technology enables head mounted devices to interact with users based on the users' eye movement or eye orientation. Typically, the factory will calibrate the head mounted device before its release to customers. Once home with the customer or user, however, the product may be impacted by various mechanical or thermal effects. For example, during the lifetime of the products, the head mounted device may be dropped or shaken or exposed to various temperature changes, e.g., seasonal changes such as winter to summer.
Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.
Embodiments of a head mounted device and a calibration apparatus removably coupled to the head mounted device are described herein. In aspects, the calibration apparatus assists in calibrating an eye tracking (ET) camera of the head mounted device. In aspects, the calibration apparatus includes a plurality of 3D objects, e.g., cylinders or pillars having varying heights which are located at varying distances from each other.
In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In some implementations of the disclosure, the term “near-eye” may be defined as including an element that is configured to be placed within 50 mm of an eye of a user while a near-eye device is being utilized. Therefore, a “near-eye optical element” or a “near-eye system” would include one or more elements configured to be placed within 50 mm of the eye of the user.
In aspects of this disclosure, visible light may be defined as having a wavelength range of approximately 380 nm-700 nm. Non-visible light may be defined as light having wavelengths that are outside the visible light range, such as ultraviolet light and infrared light. Infrared light having a wavelength range of approximately 700 nm-1 mm includes near-infrared light. In aspects of this disclosure, near-infrared light may be defined as having a wavelength range of approximately 700 nm-1.6 μm.
In aspects of this disclosure, the term “transparent” may be defined as having greater than 90% transmission of light. In some aspects, the term “transparent” may be defined as a material having greater than 90% transmission of visible light.
As noted above, eye tracking (ET) is becoming increasingly essential for the function of many applications and features of head mounted devices. ET performance is highly sensitive to proper calibration based on the positions of camera modules and light sources. During regular operation or transport, however, the head mounted device may be subject to movement and environmental forces. Accordingly, the head mounted device may need to be re-calibrated at intervals to achieve the same level of performance as right after the factory calibration. Unfortunately, it is not practical to send it back to the factory for recalibration.
Accordingly, a head mounted device that is removably coupled to a calibration apparatus is described herein. In some aspects, the head mounted device can be calibrated during charging and may be provided as an accessory with the head mounted device. In aspects, an ET camera of the head mounted device can be recalibrated periodically, enabling a similarly high level of performance of the head mounted device to its factory calibration. In aspects, the calibration apparatus includes a plurality of cylinder-shaped 3D objects or pillars having varying heights which are located at varying distances from each other. These and other embodiments are described in more detail in connection with
To further illustrate,
In some examples, a radius of a sphere or some reflective geometries of 3D ball 205 (see r-ball 205A) may be in a range of 8-10 millimeters (mm) and a plane of 3D ball surfaces may be approximately 12 millimeters away from a nominal eye position or frame center (discussed in more detail with respect to
In some examples, diameters of cylinders (d-cyl 202A) may be in a range of 0.2-1 mm. In various aspects, top surfaces of cylinders 202 may be various distances from a backside plane of frame 206, ranging from 8-12 mm. In some aspects, an average distance between a lens and the calibration apparatus is an eye relief distance of 4.5 mm-20 mm. In the example, distances between two neighboring cylindrical patterns (p-cyl) range from 2-3 mm. In some embodiments, dimension tolerance of these features may be less than 0.05 millimeters and a total number of the 3D objects or calibration targets may vary from 20×20 or 30×20 per eye or lens. Note that these ranges are merely examples and any suitable ranges that can mimic positions of virtual objects to facilitate calibration of an ET camera are contemplated.
In embodiments, process 400 may be performed by head mounted device 300 or elements of head mounted device 300 discussed further below. At a first process block 405, in some embodiments, an ET camera (e.g., 347A/347B) of
The image of the 3D pillars or cylinders is also used to determine an updated position of the ET camera (sometimes referred to as extrinsic calibration).
In aspects, a position of ET camera 347A or 347B includes an ET camera's position relative to a projected nominal eye position or a frame center. In embodiments, the frame center (e.g., 388) is defined as where an eyeball may sit relative to a frame edge, e.g., 307, when aligned with an optical axis of lens 301.
At a next process block 410, light is projected onto calibration apparatus 308 including the plurality of second objects, 3D balls 305, by LED's 309. In aspects, at next process block 415, the ET camera takes a second image of 3D balls 305 to collect second calibration information including a location of glints on 3D balls 305. In embodiments, the second calibration information can be used to calculate, using geometric optics, a position of one or more of the plurality of light sources, LED's 309.
Thus, at a process block 420, the first and second camera calibration information is used to generate updated parameters for an eye tracking computation. The first camera calibration information includes an image of the plurality of 3D pillars or cylinders. In aspects, the updated camera model information is used together with locations of reflections or glints on 3D balls to determine where LED's 309 are located. Multiple simultaneous images from the cameras along with the different pillars can also be used together to validate the calibrations. In some aspects, the location is a virtual location due to layers in a lens 301 and a location of LEDs 309. In embodiments, process 400 enables new calibration parameters to allow an ET tracking algorithm to compute or re-predict where a user's gaze or pupil is located.
As noted above, processing logic may cause example head mounted device 300 to perform above process 400. In aspects, one or more memories is coupled to the processing logic, the one or more memories storing instructions that when executed by the processing logic, cause head mounted device 300 to perform operations as described above. For simplification, only one lens (a right-eye lens) has been shown. It is understood that a left eye lens of head mounted device 300 may also include another calibration apparatus to calibrate ET cameras associated with the left eye lens.
The order in which some or all of the process blocks appear in process 400 should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel.
Light sources 560 may be, for example, light emitting diodes (LEDs), vertical-cavity surface-emitting lasers (VCSELs), micro light emitting diode (micro-LED), an edge emitting LED, a superluminescent diode (SLED), or another type of light source. Light sources 560 emit non-visible light, according to an embodiment. Light sources 560 emit near infrared light, according to an embodiment. Infrared light from other sources may illuminate the eye as well. During operation, reflected light from light sources 560 is reflected off of a user's eye and is received by ET cameras 547A and 547B. In aspects, during calibration, reflected light from light sources 560 create glints on 3D balls 505 that are captured in an image taken by camera 547A and 547B during calibration process 400.
As illustrated, head mounted device 500 includes frame 506 coupled to arms 504A and 504B. Lenses 501A and 501B are mounted to frame 506. Lenses 501A and 501B may appear transparent to a user to facilitate augmented reality or mixed reality to enable a user to view scene light from the environment around her while also receiving image light directed to her eye(s) by, for example, waveguides 550A and 550B. In some aspects, waveguides 550A/550B direct image light generated by a display 530A/530B to an eyebox area for viewing by a user of head mounted device 500. Display 530A/530B may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, micro-LED display, quantum dot display, pico-projector, or liquid crystal on silicon (LCOS) display for directing image light to a wearer of head mounted device 500.
Note that ET cameras 547A or 547B may include a complementary metal-oxide semiconductor (CMOS) image sensor. Note also that although two ET cameras have been illustrated on the head mounted devices in the FIGS., the number as well as location or position of ET cameras may vary according to features offered by the relevant AR or VR system. Note also that head mounted device 500 may detect a need for calibration and proceed with calibration when not in use or during charging. Charging station 590 may provide a stable platform or cradle for calibration. Charging station 590 may provide wireless or wired power and/or may be configured to be powered by one or more batteries.
In aspects, head mounted device 500 may include supporting hardware incorporated into frame 506 and/or its temple arms (e.g., 504A/504B). The hardware of head mounted device 100 may include, e.g., a controller 518, that may assist in performance of calibration of ET cameras 547A/547B. Controller 518 and/or a processing logic 520 may include circuitry, logic, ASIC circuitry, FPGA circuity, and/or one or more processors. Controller 518 may include any of processing logic 520, wired, and/or wireless data interface for sending and receiving data, graphic processors, and one or more memories 522 for storing data and computer executable instructions. One or more memories 522 may store instructions that when executed by processing logic 520, cause head mounted device 500 to perform operations as described in process 400 of
Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head mounted device connected to a host computer system, a standalone head mounted device, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
A “memory” or “memories” (e.g. 522) described in this disclosure may include one or more volatile or non-volatile memory architectures. The “memory” or “memories” may be removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disks (DVD), high-definition multimedia/data storage disks, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.
A Network that an example head mounted device may be connected to may include any network or network system such as, but not limited to, the following: a peer-to-peer network; a Local Area Network (LAN); a Wide Area Network (WAN); a public network, such as the Internet; a private network; a cellular network; a wireless network; a wired network; a wireless and wired combination network; and a satellite network.
Communication channels may include or be routed through one or more wired or wireless communication utilizing IEEE 802.11 protocols, short-range wireless protocols, SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), USB (Universal Serial Port), CAN (Controller Area Network), cellular data protocols (e.g. 3G, 4G, LTE, 5G), optical communication networks, Internet Service Providers (ISPs), a peer-to-peer network, a Local Area Network (LAN), a Wide Area Network (WAN), a public network (e.g. “the Internet”), a private network, a satellite network, or otherwise.
A computing device may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise. A server computer may be located remotely in a data center or be stored locally.
The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible or non-transitory machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit (“ASIC”) or otherwise.
A tangible non-transitory machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-readable storage medium includes recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).
The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.
These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.
