Patent Application
20230280468
The present disclosure generally relates to depth sensing, and specifically relates to an addressable projector for dot based direct time-of-flight depth sensing.
To achieve simple calibration and robust transmission/reception alignment for the solid state dot array illumination and direct time-of-flight (dToF) based depth sensing, it is critical to maintain the alignment of projected dots and their corresponding groups of pixels (i.e., unit cells) at a detector to ensure that a detected depth is matched to its corresponding dot (beam) within a defined work range. However, due to a specific baseline between a transmitter (i.e., projector or illuminator) and a receiver (i.e., detector) and various tolerances in constituent components of the transmitter and the receiver, substantial shift of an emitted dot with respect to its targeted unit cell typically occurs within the defined working range. This leads to cross-talk and ambiguity in depth reconstruction between neighboring unit cells at the detector.
There are currently several approaches for addressing the cross-talk and ambiguity in depth reconstruction. One approach is to reduce projection dot density aggressively to meet the specific transmitter/receiver alignment requirement. However, the sparse dot pattern impacts depth densification performance. Another approach is to use active alignment of the constituent components in the transmitter and receiver. However, the depth sensing performance are typically limited by the design of these constituent components. Yet another approach is to reduce a transmitter/receiver baseline. However, it is typically very challenging to reduce the baseline because of possible achievable sizes of the transmitter and receiver. Yet another approach is to tighten tolerances in the transmitter and/or the receiver to achieve less variance in optical alignment. However, this may lead to low yield in mass production and therefore high cost of depth sensing devices. Yet another approach is to reduce a projection dot size and reduce a dot blurring due to a reception lens (e.g., diffraction limited lens). However, this approach is typically less effective when used in isolation from other projection schemes.
A projector for illuminating a target area is presented herein that mitigates cross-talk and ambiguity in depth reconstruction. The projector includes an array of emitters having a plurality of subarrays and an optical assembly. Each subarray includes one or more independently addressable channels emitting light in accordance with emission instructions. At least two of the subarrays are adjacent to each other and do not overlap. The optical assembly is configured to tile portions of the emitted light to form a light pattern for projection to a target area. The light pattern has a first plurality of sections and a second plurality of sections, each section of the first plurality representing a first respective portion of the light pattern emitted from a corresponding subarray, and each section of the second plurality representing a second respective portion of the light pattern formed by tiling light emitted from two or more of the subarrays. In some embodiments, the projector is part of a depth camera assembly configured to determine depth information associated with one or more objects in the target area, e.g., in a local area surrounding the projector. In some other embodiments, the projector is part of an eye tracker that determines information about gaze direction for an eye.
In some embodiments, a headset includes the projector. The headset further includes a display and an optical assembly. The headset may be part of an artificial reality system. The display is configured to emit image light. The optical assembly is configured to direct the image light to an eye-box of the headset corresponding to a location of a user’s eye. The image light may comprise the depth information of the one or more objects in the local area determined by the headset. In one embodiment, the headset is implemented as an eyeglass-type platform representing a near-eye display. In another embodiment, the headset is implemented as a head-mounted display.
The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.
Addressable dot projectors are among those projectors that make use of the addressable near infra-red (NIR) vertical cavity surface emitting laser (VCSEL) array technology that is superior to the edge-emitting laser diodes. Addressable VCSEL dot emitters can be used in combination with one or more optical elements (e.g., a diffractive beam splitting optical element) to increase the number of dots in the projection.
Embodiments of the present disclosure relate to a structure and operation of an addressable projector that enables high performance dot-based direct time-of-flight (dToF) depth sensing. The addressable projector presented herein has a flexible configurability, higher tolerance for a misalignment between different components of a depth sensing apparatus (e.g., between the projector and the detector), lower beam divergence, etc. Furthermore, the addressable projector presented herein has a small form factor.
The addressable projector presented herein may be part of a system for, e.g., dot-based dToF depth sensing. A proper alignment between a transmitter (i.e., the addressable projector) and a receiver (i.e., the detector) of the depth sensing system can be achieved through time multiplexing of sparse light patterns with a simple emitter array addressing and pattern tiling. By time multiplexing the addressable tiled patterns, a denser depth point cloud can be effectively obtained while each addressable tiled pattern achieves a robust alignment to its corresponding unit cells at the detector, with the advantage of reduced (or, in some embodiments, eliminated) detection cross-talk and/or ambiguity between neighboring unit cells at the detector.
The addressable projector presented herein includes an array of emitters (e.g., VCSEL array) and an optical assembly. The array of emitters features a simple addressing and provides an increased projection dot density while adapting to a field-dependent allowable dot density (e.g., as much dense as possible at a center of a field-of-view (FOV) and sparser toward edges/corners of the FOV). The array of emitters includes a plurality of subarrays that are adjacent to each other, and each subarray includes one or more independently addressable channels. The optical assembly is configured to tile light from the plurality of subarrays to form a dot pattern, of a plurality of dot patterns, in a local area of the projector. The dot pattern has a dot density higher in a center of the dot pattern than at a periphery of the dot pattern. The dot pattern is determined in part by which of the one or more channels are active in each of the plurality of subarrays.
In some embodiments, the addressable projector is part of a depth camera assembly (DCA) that determines depth information for one or more objects in a target area (e.g., a portion of the local area surrounding the DCA) based on light reflected from the one or more objects, the light being emitted from the addressable projector. In some other embodiments, the addressable projector is part of an eye tracker configured to determine a gaze direction for an eye based on light reflected from at least one surface of the eye, the light being emitted from the addressable projector.
Embodiments of the present disclosure may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a headset, a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a near-eye display (NED), a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
In some embodiments, a headset can integrate the projector as part of the DCA. Alternatively or additionally, the headset may integrate the projector as part of the eye tracker. The headset further includes a display and an optical assembly. The headset may be part of an artificial reality system. The display is configured to emit image light. The optical assembly is configured to direct the image light to an eye-box of the headset corresponding to a location of a user’s eye. The image light may comprise the depth information of the one or more objects in the local area determined by the headset. In one embodiment, the headset is implemented as an eyeglass-type platform representing a NED. In another embodiment, the headset is implemented as a HMD.
The headset 100 may correct or enhance the vision of a user, protect the eye of a user, or provide images to a user. The headset 100 may be a NED that produces artificial reality content for the user. The headset 100 may be eyeglasses which correct for defects in a user’s eyesight. The headset 100 may be sunglasses which protect a user’s eye from the sun. The headset 100 may be safety glasses which protect a user’s eye from impact. The headset 100 may be a night vision device or infrared goggles to enhance a user’s vision at night.
The frame 105 holds the other components of the headset 100. The headset 100 includes a front part that holds the one or more display elements 110 and end pieces to attach to a head of the user. The front part of the frame 105 bridges the top of a nose of the user. The end pieces (e.g., temples) are portions of the frame 105 to which the temples of a user are attached. The length of the end piece may be adjustable (e.g., adjustable temple length) to fit different users. The end piece may also include a portion that curls behind the ear of the user (e.g., temple tip, ear piece).
The one or more display elements 110 provide light to a user wearing the headset 100. As illustrated, the headset 100 includes a display element 110 for each eye of a user. In some embodiments, a display element 110 generates image light that is provided to an eye box of the headset 100. The eye box is a location in space that an eye of user occupies while wearing the headset 100. For example, a display element 110 may be a waveguide display. A waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is in-coupled into the one or more waveguides which outputs the light in a manner such that there is pupil replication in an eye box of the headset 100. In-coupling and/or outcoupling of light from the one or more waveguides may be done using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., waveguide, mirror, etc.) that scans light from the light source as it is in-coupled into the one or more waveguides. Note that in some embodiments, one or both of the display elements 110 are opaque and do not transmit light from a local area around the headset 100. The local area is the area surrounding the headset 100. For example, the local area may be a room that a user wearing the headset 100 is inside, or the user wearing the headset 100 may be outside and the local area is an outside area. In this context, the headset 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 110 are at least partially transparent, such that light from the local area may be combined with light from the one or more display elements to produce AR and/or MR content.
In some embodiments, a display element 110 does not generate image light, and instead is a lens that transmits light from the local area to the eye box. For example, one or both of the display elements 110 may be a lens without correction (non-prescription) or a prescription lens (e.g., single vision, bifocal and trifocal, or progressive) to help correct for defects in a user’s eyesight. In some embodiments, the display element 110 may be polarized and/or tinted to protect the user’s eyes from the sun.
Note that in some embodiments, the display element 110 may include an additional optics block (not shown). The optics block may include one or more optical elements (e.g., lens, Fresnel lens, etc.) that direct light from the display element 110 to the eye box. The optics block may, e.g., correct for aberrations in some or all of the image content, magnify some or all of the image, or some combination thereof.
The DCA determines depth information for a portion of a local area surrounding the headset 100. The DCA includes one or more imaging devices 115, a projector 120, and a DCA controller (not shown in
The DCA controller computes depth information for the portion of the local area using the captured images and one or more depth determination techniques. The depth determination technique may be dToF depth sensing. Alternatively or additionally, the depth determination technique can be indirect time-of-flight (iToF) depth sensing, structured light depth sensing, passive stereo analysis based depth sensing, active stereo analysis based depth sensing (uses texture added to the scene by light from the projector 120), some other technique to determine depth of a scene, or some combination thereof.
Based on the determined depth information, the DCA controller may determine absolute positional information of the headset 100 within the local area. The DCA controller may also generate a model of the local area. The one or more imaging devices 115 may be integrated with the headset 100 or may be positioned within the local area external to the headset 100. In some embodiments, the DCA controller may provide the depth image data to the headset controller 125 integrated into the headset 100, e.g., for further processing and/or communication to some other component of an artificial reality system that includes the headset 100. The one or more imaging devices 115 may be part of simultaneous localization and mapping (SLAM) sensors mounted on the headset 100 for capturing visual information of a local area surrounding some or all of the headset 100.
The headset controller 125 may control operations of one or more components of the headset 100 including the projector 120. The headset controller 125 may receive the depth image data from the DCA controller and perform additional processing on the depth image data. In some embodiments, the headset controller 125 may control operations of components of an audio system integrated into the headset 100 (not shown in
The position sensor 130 generates one or more measurement signals in response to motion of the headset 100. The position sensor 130 may be located on a portion of the frame 105 of the headset 100. The position sensor 130 may include a position sensor, an inertial measurement unit (IMU), or both. Some embodiments of the headset 100 may or may not include the position sensor 130 or may include more than one position sensors 130. In embodiments in which the position sensor 130 includes an IMU, the IMU generates IMU data based on measurement signals from the position sensor 130. Examples of position sensor 130 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU, or some combination thereof. The position sensor 130 may be located external to the IMU, internal to the IMU, or some combination thereof.
Based on the one or more measurement signals, the position sensor 130 estimates a current position of the headset 100 relative to an initial position of the headset 100. The estimated position may include a location of the headset 100 and/or an orientation of the headset 100 or the user’s head wearing the headset 100, or some combination thereof. The orientation may correspond to a position of each ear relative to a reference point. In some embodiments, the position sensor 130 uses the depth information and/or the absolute positional information from the DCA to estimate the current position of the headset 100. The position sensor 130 may include multiple accelerometers to measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes to measure rotational motion (e.g., pitch, yaw, roll). In some embodiments, an IMU rapidly samples the measurement signals and calculates the estimated position of the headset 100 from the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on the headset 100. The reference point is a point that may be used to describe the position of the headset 100. While the reference point may generally be defined as a point in space, however, in practice the reference point is defined as a point within the headset 100.
The display assembly 140 is configured to direct the image light to the eye 155 through the eye box 150. In some embodiments, when the headset 100 is configured as an AR NED, the display assembly 140 also directs light from a local area surrounding the headset 100 to the eye 155 through the eye box 150. The display assembly 140 may be configured to emit image light at a particular focal distance in accordance with varifocal instructions, e.g., provided from a varifocal module (not shown in
The display assembly 140 may be composed of one or more materials (e.g., plastic, glass, etc.) with one or more refractive indices that effectively minimize the weight and present to the user a field of view of the headset 100. In alternate configurations, the headset 100 includes one or more optical elements between the display assembly 140 and the eye 155. The optical elements may act to, e.g., correct aberrations in image light emitted from the display assembly 140, magnify image light, perform some other optical adjustment of image light emitted from the display assembly 140, or some combination thereof. The example for optical elements may include an aperture, a Fresnel lens, a convex lens, a concave lens, a liquid crystal lens, a diffractive element, a waveguide, a filter, a polarizer, a diffuser, a fiber taper, one or more reflective surfaces, a polarizing reflective surface, a birefringent element, or any other suitable optical element that affects image light emitted from the display assembly 140.
The frame 105 further includes the DCA 145 configured to determine depth information of one or more objects in a local area surrounding some or all of the headset 100. For purposes of illustration,
The projector 120 may be configured to illuminate a target area (e.g., at least a portion of the local area) with a light pattern in accordance with emission instructions generated by the DCA controller 160. The projector 120 may include an array of emitters having a plurality of subarrays. Each subarray of the projector 120 may include one or more independently addressable channels emitting light in accordance with the emission instructions, and at least two of the subarrays may be adjacent to each other and may not overlap. In one embodiment, the projector 120 includes two independently addressable subarrays that do not overlap. In another embodiment, the projector 120 includes four independently addressable subarrays that do not overlap. In some other embodiments, the projector 120 includes multiple independently addressable subarrays that at least partially overlap. Examples of addressing projector layouts are shown in
The light emitted from the array of emitters of the projector 120 may be projected into the target area as a light pattern by an optical assembly of the projector 120 (not shown in
The imaging device 115 includes one or more cameras configured to capture one or more images of at least a portion of the light reflected from one or more objects in the local area. In one embodiment, the imaging device 115 is an infrared camera configured to capture images in a NIR spectrum. Additionally, the imaging device 115 may be also configured to capture images of visible spectrum light. The imaging device 115 may include a charge-coupled device (CCD) detector, a complementary metal-oxide-semiconductor (CMOS) detector, a single-photon avalanche diodes (SPAD) detector, some other types of detector, or combination thereof. The imaging device 115 may be configured to operate with a frame rate in the range of approximately 30 Hz to approximately 1 KHz for fast detection of objects in the local area. In some embodiments, the imaging device 115 is deactivated for a defined amount of time before being activated again. Alternatively or additionally, the imaging device 115 can operate as instructed by the DCA controller 160 for single or multiple frames, up to a maximum frame rate, which can be in the kilohertz range.
The DCA controller 160 may generate the emission instructions and provide the emission instructions to the projector 120 for controlling operation of at least a portion of emitters in the array in the projector 120 to emit light. The DCA controller 160 may control, based on the emission instructions, operation of the projector 120 to dynamically adjust a pattern of the light illuminating the local area, an intensity of the light pattern, a density of the light pattern, location of the light pattern being projected at the target area, combination thereof, etc. The DCA controller 160 may be also configured to determine depth information for the one or more objects in the target area based in part on the one or more images captured by the imaging device 115. In some embodiments, the DCA controller 160 provides the determined depth information to a console (not shown in
In some embodiments, the headset 100 further includes an eye tracker (not shown in
The emitter array 215 may include a VCSEL array, an array of light emission diodes (LEDs), some other type of emitters, or some combination thereof. Each emitter (e.g., each VCSEL) in the emitter array 215 may emit a light beam of, e.g., a circular shape having a specific level of brightness based on emission instructions from the controller 225. The size of emitter array 215 can be appropriately selected in accordance with embodiments illustrated in
The optical assembly 220 may process at least portions of light emitted from the emitter array 215 to form a light pattern (e.g., dot pattern) for projection to the target area. The optical assembly 220 may include a diffractive optical element (DOE) beam splitter, collimation optics, one or more projection lenses, some other optical element(s), or some combination thereof. The collimation optics of the optical assembly 220 may collimate at least a portion of light beams emitted from the emitter array 215 to form at least a portion of the light pattern for projection to the target area. In some embodiments, the DOE beam splitter of the optical assembly 220 (e.g., in optical series with the collimation optics) may split a light beam emitted from a corresponding emitter in the emitter array 215 to generate multiple light beams. In some other embodiments, the optical assembly 220 provides direct projection of the light emitted from the emitter array 215 using the one or more projection lenses. In such case, a power of each light beam (e.g., of a circular shape) emitted from a respective emitter (e.g., a respective VCSEL) in the emitter array 215 may be determined by the controller 225 that controls operation of the respective emitter. In both cases, the optical assembly 220 may effectively use at least portions of the light emitted from the emitter array 215 to form the light pattern for projection to the target area.
The controller 225 may control operations of the emitter array 215 (e.g., operation of the VCSEL array). The controller 225 may be coupled to each group of emitters (e.g., group of VCSELs) in the emitter array 215 for independently controlling light emission from each group of emitters. The controller 225 may be implemented as a processor implemented on the submount 212 that generates appropriate electrical signals for controlling operations of each group of emitters in the emitter array 215. The controller 225 may be also coupled to a detector 235 of the imaging device 210 for receiving light information reflected from the target area captured by the detector 235. The controller 225 may be an embodiment of the DCA controller 160.
The imaging device 210 may capture portions of the light pattern reflected from one or more objects in the target area. The imaging device 210 may include an optical assembly 230 and the detector 235. The optical assembly 230 may propagate the portions of the reflected light pattern and direct the portions of the reflected light pattern to the detector 235. The optical assembly 230 may include one or more projection lenses. The detector 235 captures light intensities in relation to the portions of the reflected light pattern. The detector 235 may include an array of a single-photon avalanche diodes (SPADs), or some other array of sensing elements capable of capturing multiple light intensities substantially at the same time instant. In some embodiments, each sensing element (e.g., each SPAD) of the detector 235 may capture a light intensity for a respective light beam (e.g., of a circular shape or dot) reflected from a respective portion of the target area. Light signals from the detector 235 (e.g., a photon arrival time in the form of time-to-digital histogram, or some other light information) may be provided to a controller (e.g., a controller of the imaging device 210 or the controller 225) for determination of depth information for the target area based in part on the light information captured at the detector 235.
Each emitter (e.g., each VCSEL) in the emitter array 305 may include a respective beam-shaping optics 320 (e.g., diffractive optical element or metasurface) that emits a respective light beam of a particular shape (e.g., circular shape). The beam-shaping optics 320 may comprise one or more sub-wavelength optical structures configured to change an amplitude, phase and/or polarization of the incident light. Each VCSEL in the emitter array 305 may further include a respective anode 325 coupled to the beam-shaping optics 320. The anode 325 is an electrode where electricity moves into (e.g., based on a voltage or current signal from the controller 310). The electricity from the anode 325 flows into an emitter cavity 330 where output light is generated, which then undergoes beam shaping when passing through the beam-shaping optics 320. One or more anodes 325 may be coupled to a corresponding cathode 335 connected to the emitter array 305 over a cathode via 340. A desired addressability of the emitter array 305 can be achieved by properly grouping anodes 325, or both anodes 325 and cathodes 335.
The optical assembly of the projector having the addressing layout 405 may tile light from the two subarrays of emitters A and B into the tiled illumination 410, as shown in
The tiled illumination 410 has an illumination density higher in a center part of a FOV than at a peripheral part of the FOV. This may lead to, e.g., appropriately denser dots in the center part of the FOV and sparser dots towards the peripheral part of the FOV that typically exhibits worst alignment between the projector and a detector (e.g., the detector 235) and has less importance in depth sensing. At the same time, the emitter array (e.g., VCSEL array) of the projector having the addressing layout 405 can be implemented with relatively simple addressing and routing, while having viability for high-speed operations.
The projector having the addressing layout 405 may be utilized along with the detector for time multiplexed pattern projection and detection. In one or more embodiments, two sub-frames of light illumination and data capturing is used to generate one frame of depth sensing data. For example, during a first sub-frame, only the subarray A may be activated so that the projector illuminates a target area with the light pattern A′. During the first sub-frame, the detector may activate a first portion of pixels corresponding to the subarray A and output first depth sensing data. The first subframe may be followed by a second sub-frame during which only the subarray B is activated so that the projector illuminates the target area with the light pattern B′. Reflected portions of the light pattern B′ may be detected by a second portion of pixels of the detector corresponding to the subarray B, and the detector may output second depth sensing data. The first and second depth sensing data may be fused/combined (e.g., at the controller 225) to output a single frame of depth sensing data.
The optical assembly of the projector having the addressing layout 415 may tile light from the four subarrays of emitters A, B, C and D into the tiled illumination 420, as shown in
Similarly as for the tiled illumination 410, the tiled illumination 420 represents a light pattern with two types of sections. Each section of the first type (e.g., section A′, section B′, section C′, and section D′) represents a first respective portion of the light pattern emitted from a single corresponding subarray of emitters, i.e., from the subarray A, the subarray B, the subarray C, or the subarray D. Each section of the second type (e.g., sections A′/B′, A′/C′, B′/D′, C′/D′, A′/B′/C′/D′) represents a second respective portion of the light pattern formed by tiling and overlapping light emitted from multiple subarrays (e.g., two or four subarrays).
Again, similarly as for the tiled illumination 410, the tiled illumination 420 has an illumination density higher in a center part of a FOV than at peripheral parts of the FOV. This may lead to, e.g., appropriately denser dots in the center part of the FOV and sparser dots towards the peripheral parts of the FOV that typically exhibits worst alignment between the projector and the detector and has less importance in depth sensing. At the same time, the emitter array (e.g., VCSEL array) of the projector having the addressing layout 415 can be implemented with relatively simple addressing and routing, while having viability for high-speed operations.
Note that tiling examples in
The optical assembly of the projector having the addressing layout 505 may tile light from the four subarrays into the projection pattern 510, as shown in
Each circle in the projection pattern 510 having a corresponding pattern fill represents a light beam that originates from a corresponding independently addressable channel of the addressing layout 505. Note that multiple beams in the projection pattern 510 correspond to a single emitter in the addressing layout 505. In some embodiments, each circle in the projection pattern 510 may also correspond to an imaged position (i.e., pixel, memory element, or unit cell) on a detector of an imaging device (e.g., the detector 235).
The optical assembly of the projector having the addressing layout 515 may tile light from the four subarrays into the projection pattern 520, as shown in
Each circle in the projection pattern 520 having a corresponding pattern fill represents a light beam that originates from a corresponding independently addressable channel of the addressing layout 515. Note that multiple beams in the projection pattern 520 correspond to a single emitter in the addressing layout 515. In some embodiments, each circle in the projection pattern 520 may also correspond to an imaged position (i.e., pixel, memory element, or unit cell) on a detector of an imaging device (e.g., the detector 235).
The optical assembly of the projector having the addressing layout 515 may tile light from the four subarrays into the projection pattern 530, as shown in
Each circle in the projection pattern 530 having a corresponding pattern fill represents a light beam that originates from a corresponding independently addressable channel of the addressing layout 525. Note that multiple beams in the projection pattern 530 correspond to a single emitter in the addressing layout 525. In some embodiments, each circle in the projection pattern 530 may also correspond to an imaged position (i.e., pixel, memory element, or unit cell) on a detector of an imaging device (e.g., the detector 235).
The projector receives 605 emission instructions specifying activation of one or more independently addressable channels in each subarray of a plurality of subarrays of an array of emitters, at least two of the subarrays being adjacent to each other and do not overlap. Each emitter of the array of emitters may be a VCSEL. The array of emitters may comprise a pair of subarrays position side by side to each other (e.g., illustrated in
The projector activates 610 the specified one or more channels in each subarray based on the emission instructions. The projector may control operation of each subarray by activating or deactivating each of the one or more channels in that subarray, based at least in part on the emission instructions.
The projector illuminates 615 a target area with a light pattern formed by tiling light emitted from the activated one or more channels in each subarray. The light pattern may be composed of a first plurality of sections and a second plurality of sections. Each section of the first plurality may represent a first respective portion of the light pattern emitted from a corresponding subarray. Each section of the second plurality may represent a second respective portion of the light pattern formed by tiling light emitted from two or more of the subarrays. The light pattern emitted from the array of emitters may be a dot pattern. The light pattern may have an illumination density higher in a center of the light pattern than at a periphery of the light pattern. The light pattern may be determined in part by which of the one or more channels are active in each of the subarrays. The light pattern may comprise a pattern of circular shapes, where each shape may be defined by an emission area (e.g., non-circular emission area) of a corresponding emitter within the array of emitters.
In some embodiments, the projector is part of a DCA that includes an imaging device and a controller. The imaging device captures one or more images of at least a portion of the light pattern reflected from one or more objects in the target area. The controller generates the emission instructions, provides the emission instructions to the projector, and determines depth information for the one or more objects based in part on the captured one or more images. In some embodiments, a detector of the imaging device in the DCA is instructed to activate a group of pixels (or unit cells) of the detector corresponding to the projected light pattern, and to output light signals captured by the activated group of pixels. The controller may determine the depth information for the one or more objects in the target area using the light signals captured by the activated group of pixels and output by the detector. The DCA may provide the determined depth information to a console coupled to the headset that generates content for presentation on a display of the headset, based on the depth information. Alternatively, the DCA may provide the determined depth information to a module of the headset that generates content for presentation on the display of the headset, based on the depth information.
The DCA with the projector presented herein may be integrated into a headset as part of an AR system. In this case, the headset may be configured to sense and display objects behind a head of a user wearing the headset or display objects recorded previously. Alternatively, the DCA with the projector may be integrated into a base station or a sensor bar external to the headset. In this case, the DCA may be configured to sense various body parts of a user wearing the headset or portions of an environment, e.g., for recording the geometry of a user or a local area, hand-tracking, eye-tracking, face recognition, gesture recognition, environment scanning, environment reconstruction, etc. In some embodiments, the DCA with the projector is integrated into a portable computing platform (e.g., a mobile phone or tablet).
In some other embodiments, the projector presented herein is part of an eye tracker, and the light pattern from the projector illuminates the target area that includes an eye box of a headset. An imaging device of the eye tracker captures one or more images of at least a portion of the light pattern reflected from at least one surface of an eye in the eye box. A controller of the eye tracker determines position and orientation of the eye in the eye box (e.g., gaze direction for eye) based on the captured one or more images.
The headset 705 is a NED or a HMD that presents content to a user comprising virtual and/or augmented views of a physical, real-world environment with computer-generated elements (e.g., two-dimensional or three-dimensional images, two-dimensional or three-dimensional video, sound, etc.). In some embodiments, the presented content includes audio that is presented via an external device (e.g., speakers and/or headphones) that receives audio information from the headset 705, the console 710, or both, and presents audio data based on the audio information. The headset 705 may comprise one or more rigid bodies, which may be rigidly or non-rigidly coupled together. A rigid coupling between rigid bodies causes the coupled rigid bodies to act as a single rigid entity. In contrast, a non-rigid coupling between rigid bodies allows the rigid bodies to move relative to each other. An embodiment of the headset 705 is the headset 100 of
The headset 705 may include a display 720, an optics block 725, one or more position sensors 730, an IMU 735, a DCA 740, and a headset controller 750. Some embodiments of the headset 705 have different and/or additional components than those described in conjunction with
The display 720 displays two-dimensional or three-dimensional images to the user in accordance with data received from the console 710. In various embodiments, the display 720 comprises a single display or multiple displays (e.g., a display for each eye of a user). Examples of the display 720 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an inorganic light emitting diode (ILED) display, an active-matrix organic light-emitting diode (AMOLED) display, a transparent organic light emitting diode (TOLED) display, a laser-based display, one or more waveguides, some other display, a scanner, one-dimensional array, or some combination thereof. Content displayed on the display 720 may include the depth information determined by the DCA 740. An embodiment of the display 720 is the display assembly 140.
The optics block 725 magnifies image light received from the display 720, corrects optical errors associated with the image light, and presents the corrected image light to a user of the headset 705. In various embodiments, the optics block 725 includes one or more optical elements. Example optical elements included in the optics block 725 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflecting surface, or any other suitable optical element that affects image light. Moreover, the optics block 725 may include combinations of different optical elements. In some embodiments, one or more of the optical elements in the optics block 725 may have one or more coatings, such as partially reflective or anti-reflective coatings.
Magnification and focusing of the image light by the optics block 725 allows the display 720 to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification may increase the field of view of the content presented by the display 720. For example, the field of view of the displayed content is such that the displayed content is presented using almost all (e.g., approximately 110 degrees diagonal), and in some cases all, of the user’s field of view. Additionally, in some embodiments, the amount of magnification may be adjusted by adding or removing optical elements.
In some embodiments, the optics block 725 may be designed to correct one or more types of optical error. Examples of optical error include barrel or pincushion distortion, longitudinal chromatic aberrations, or transverse chromatic aberrations. Other types of optical errors may further include spherical aberrations, chromatic aberrations, or errors due to the lens field curvature, astigmatisms, or any other type of optical error. In some embodiments, content provided to the electronic display for display is pre-distorted, and the optics block 725 corrects the distortion when it receives image light from the electronic display generated based on the content.
The IMU 735 is an electronic device that generates data indicating a position of the headset 705 based on measurement signals received from one or more of the position sensors 730. A position sensor 730 generates one or more measurement signals in response to motion of the headset 705. Examples of position sensors 730 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU 735, or some combination thereof. The position sensors 730 may be located external to the IMU 735, internal to the IMU 735, or some combination thereof. An embodiment of the position sensor 735 is the position sensor 130.
The DCA 740 includes a projector 741, one or more imaging devices 743 and a DCA controller 745. The DCA 740 generates depth image data of a local area surrounding some or all of the headset 705. Depth image data includes pixel values defining distance from the imaging device, and thus provides a (e.g., 3D) mapping of locations captured in the depth image data. An embodiment of the DCA 740 is the DCA 145, an embodiment of the projector 741 is the projector 120, an embodiment of the imaging device 743 is the imaging device 115, and an embodiment of the DCA controller 745 is the DCA controller 160.
In some embodiments, the DCA 740 generates depth image data using the structured light depth sensing technique. The DCA 740 may emit structured light and determine depth information for the local area by capturing reflected and deformed structured light pattern. In some other embodiments, the DCA 740 generates depth image data using the time-of-flight depth sensing technique. The DCA 740 may generate the depth image data based on time required to light to be emitted from the projector 741 until at least a portion of the light reflected from one or more objects in the local area is captured by the one or more imaging devices 743. In some other embodiments, the DCA 740 generates depth image data using active or passive stereo imaging. By comparing information about a local area from two vantage points, the DCA 740 may extract depth information based on relative positions of objects in two panels.
The projector 741 may be configured to illuminate a target area (e.g., at least a portion of the local area) with a light pattern in accordance with emission instructions generated by the DCA controller 745. The projector 741 may include an array of emitters having a plurality of subarrays. Each subarray of the projector 741 may include one or more independently addressable channels emitting light in accordance with emission instructions, and at least two of the subarrays may be adjacent to each other and do not overlap. At least the portion of the emitters in the array of the projector 741 may emit light in the NIR spectrum. Each emitter in the array of emitters in the projector 741 may be implemented as a VCSEL emitter that emits in, e.g., the NIR spectrum. The light emitted from the array of emitters of the projector 741 may be projected into the target area as a light pattern by an optical assembly of the projector 741. The optical assembly of the projector 741 may be implemented as a diffractive beam splitting optical element, a collimation optical element, one or more projection lenses, some other type of optical elements, or some combination thereof. The optical assembly of the projector 741 may tile portions of the emitted light to form the light pattern for projection to the target area. The light pattern emitted by the projector 741 may be composed of a first plurality of sections and a second plurality of sections, each section of the first plurality representing a first respective portion of the light pattern emitted from a corresponding subarray, and each section of the second plurality representing a second respective portion of the light pattern formed by tiling light emitted from two or more of the subarrays. The projector 741 may be an embodiment of the projector 120, an embodiment of the projector 205, or an embodiment of the projector 300.
In some embodiments, the same type of the projector as the projector 741 can be part of an eye tracker integrated into a headset 705 (not shown in
The DCA controller 745 may generate emission instructions and provide the emission instructions to the projector 741 to control operation of at least a portion of the emitters in the projector 741. The DCA controller 745 may control operation of at least the portion of emitters in the projector 741 by activating (or deactivating) the one or more channels in each subarray based on the emission instructions. The DCA controller 745 may further generate the depth image data based on light captured by the one or more imaging devices 743 by using structed light depth sensing techniques, time-of-flight depth sensing techniques, stereo based depth sensing techniques, some other depth sensing technique, or combination thereof. The DCA controller 745 may provide the depth image data to the console 710, the headset controller 750, or some other component. In some embodiments, the DCA controller 745 controls operation of one or more emitters in the projector 741, based at least in part on the depth image data.
The I/O interface 715 is a device that allows a user to send action requests and receive responses from the console 710. An action request is a request to perform a particular action. For example, an action request may be an instruction to start or end capture of image or video data or an instruction to perform a particular action within an application. The I/O interface 715 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests and communicating the action requests to the console 710. An action request received by the I/O interface 715 is communicated to the console 710, which performs an action corresponding to the action request. In some embodiments, the I/O interface 715 includes an IMU 735 that captures calibration data indicating an estimated position of the I/O interface 715 relative to an initial position of the I/O interface 715. In some embodiments, the I/O interface 715 may provide haptic feedback to the user in accordance with instructions received from the console 710. For example, haptic feedback is provided when an action request is received, or the console 710 communicates instructions to the I/O interface 715 causing the I/O interface 715 to generate haptic feedback when the console 710 performs an action.
The console 710 provides content to the headset 705 for processing in accordance with information received from one or more of: the DCA 740, the headset controller 750, and the I/O interface 715. In the example shown in
The application store 760 stores one or more applications for execution by the console 710. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user. Content generated by an application may be in response to inputs received from the user via movement of the headset 705 or the I/O interface 715. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.
The tracking module 765 calibrates the system 700 using one or more calibration parameters and may adjust one or more calibration parameters to reduce error in determination of the position of the headset 705 or of the I/O interface 715. For example, the tracking module 765 communicates a calibration parameter to the DCA 740 to adjust the focus of the DCA 740 to more accurately determine positions of structured light elements captured by the DCA 740. Calibration performed by the tracking module 765 also accounts for information received from the IMU 735 in the headset 705 and/or an IMU included in the I/O interface 715. Additionally, if tracking of the headset 705 is lost (e.g., the DCA 740 loses line of sight of at least a threshold number of structured light elements), the tracking module 765 may re-calibrate some or all of the system 700.
The tracking module 765 tracks movements of the headset 705 or of the I/O interface 715 using information from the DCA 740, the one or more position sensors 730, the IMU 735, or some combination thereof. For example, the tracking module 765 determines a position of a reference point of the headset 705 in a mapping of a local area based on information from the headset 705. The tracking module 765 may also determine positions of the reference point of the headset 705 or a reference point of the I/O interface 715 using data indicating a position of the headset 705 from the IMU 735 or using data indicating a position of the I/O interface 715 from an IMU 735 included in the I/O interface 715, respectively. Additionally, in some embodiments, the tracking module 765 may use portions of data indicating a position or the headset 705 from the IMU 725 as well as representations of the local area from the DCA 740 to predict a future location of the headset 705. The tracking module 765 provides the estimated or predicted future position of the headset 705 or the I/O interface 715 to the engine 770.
The engine 770 generates a three-dimensional mapping of the area surrounding the headset 705 (i.e., the “local area”) based on information received from the headset 705. In some embodiments, the engine 770 determines depth information for the three-dimensional mapping of the local area based on information received from the DCA 740 that is relevant for techniques used in computing depth. The engine 770 may calculate depth information using one or more techniques in computing depth from the portion of the reflected light detected by the DCA 740, such as the stereo based techniques, the structured light illumination techniques, and the time-of-flight techniques. In various embodiments, the engine 770 uses the depth information to, e.g., update a model of the local area, and generate content based in part on the updated model.
The engine 770 also executes applications within the system 700 and receives position information, acceleration information, velocity information, predicted future positions, or some combination thereof, of the headset 705 from the tracking module 765. Based on the received information, the engine 770 determines content to provide to the headset 705 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine 770 generates content for the headset 705 that mirrors the user’s movement in a virtual environment or in an environment augmenting the local area with additional content. Additionally, the engine 770 performs an action within an application executing on the console 710 in response to an action request received from the I/O interface 715 and provides feedback to the user that the action was performed. The provided feedback may be visual or audible feedback via the headset 705 or haptic feedback via the I/O interface 715. Additional Configuration Information
The foregoing description of the embodiments of the disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.
Some portions of this description describe the embodiments of the disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.
Embodiments of the disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
Embodiments of the disclosure may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.
Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the disclosure be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the disclosure, which is set forth in the following claims.
This application claims benefit of U.S. Provisional Pat. Application Serial No. 63/315,221 filed Mar. 1, 2022, which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63315221 | Mar 2022 | US |
