Patent Grant
11438486
This Patent Application is related to the following United States utility patents: U.S. Pat. No. 9,774,801 B2, entitled “SOLID STATE IMAGE SENSOR WITH ENHANCED CHARGE CAPACITY AND DYNAMIC RANGE,” filed on Dec. 5, 2014, and assigned to the assignee hereof; and U.S. Pat. No. 9,332,200 B1, entitled “PIXEL READOUT ARCHITECTURE FOR FULL WELL CAPACITY EXTENSION,” filed on Dec. 5, 2014, and assigned to the assignee hereof. The disclosure of these patents are considered part of and are incorporated by reference in this Patent Application.
This disclosure relates generally to depth sensing systems and specifically to improving the speed and accuracy with which active depth systems generate depth information.
A device may determine distances of objects or surfaces in a scene using various active-or-passive depth sensing techniques. Passive depth sensing systems determine distances to objects in a scene based on ambient light reflected from the objects. Active depth sensing systems determine distances to objects in a scene by emitting pulses of light into a scene and analyzing corresponding light pulses reflected from objects in the scene. Some active depth sensing systems also may determine distances to objects in a scene by projecting a structured light (SL) pattern onto a scene and analyzing changes of the SL pattern reflected from objects in the scene. Active depth sensing systems typically employ either time-of-flight (ToF) techniques or SL techniques.
Some active depth sensing systems use an array of vertical cavity surface emitting lasers (VCSELs) that each emit a point of light (or “dot”) through a diffractive optical element (DOE). The DOE is often a grating-type with a periodic structure to diffract the light emitted from the VCSELs so as to form 2D array patterns (or “dot patterns”) on the scene. Some systems may then “stitch” the 2D arrays to form a pseudo-3D structured light pattern. The inherent pincushion distortion associated with such stitching may limit the resolution of the resultant image.
To eliminate this distortion and achieve higher-resolution images, an active depth sensing system may use a single DFB laser, rather than a VCSEL array, to emit a single point of light (or “dot”) through a coded DOE (rather than a grating-type DOE). The coded DOE can shape and split the single point of light so as to project a distribution of light (or “dot pattern”) onto the scene. At continuous wave or long pulse (e.g., on the order of milliseconds) operation, the light emitted from one DFB laser may have a lower power (and thus be less bright) than the light emitted from a VCSEL array, and may therefore be more susceptible to interference from ambient light (such as sunlight) than VCSEL arrays.
The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.
One innovative aspect of the subject matter described in this disclosure can be used as a method of sensing a scene. In some implementations, the method can include illuminating the scene using a sequence of two or more periods, each period including a transmission portion during which a plurality of light pulses are emitted onto the scene and including a non-transmission portion corresponding to an absence of emitted light. The method also can include receiving, during each transmission portion, a plurality of light pulses reflected from the scene, and continuously accumulating photoelectric charge indicative of the received light pulses during an entirety of the sequence. The method also can include transferring the accumulated photoelectric charge to a readout circuit after an end of the sequence.
Another innovative aspect of the subject matter described in this disclosure can be implemented in an apparatus. In some implementations, the apparatus can include a memory and a processor coupled to the memory. The processor can be configured to illuminate the scene using a sequence of two or more periods, each period including a transmission portion during which a plurality of light pulses are emitted onto the scene and including a non-transmission portion corresponding to an absence of emitted light. The processor also can be configured to receive, during each transmission portion, a plurality of light pulses reflected from the scene and to continuously accumulate photoelectric charge indicative of the received light pulses during an entirety of the sequence. The processor also can be configured to transfer the accumulated photoelectric charge to a readout circuit after an end of the sequence.
Another innovative aspect of the subject matter described in this disclosure can be implemented in a device. The device can include means for illuminating a scene using a sequence of two or more periods, each period including a transmission portion during which a plurality of light pulses are emitted onto the scene and including a non-transmission portion corresponding to an absence of emitted light. The device also can include means for receiving, during each transmission portion, a plurality of light pulses reflected from the scene, and means for continuously accumulating photoelectric charge indicative of the received light pulses during an entirety of the sequence. The device also can include means for transferring the accumulated photoelectric charge to a readout circuit after an end of the sequence.
Details of one or more implementations of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.
Like reference numerals refer to corresponding parts throughout the drawing figures.
The following description is directed to certain implementations for the purposes of describing the innovative aspects of this disclosure. However, a person having ordinary skill in the art will readily recognize that the teachings herein can be applied in a multitude of different ways. Aspects of the present disclosure relate to light projectors and include a 3D active depth sensing system that emits light in laser pulse train bursts and receives reflections of the emitted light at a gated sensor.
An active depth sensing system may emit light in a predefined distribution of points (or another suitable shape of focused light) into a scene, and the reflected light may be received by the active depth sensing system. Depths of objects in the scene may be determined by comparing the distribution of the received light and the distribution of the emitted light. In comparing the distributions, a portion of the predefined distribution for the emitted light may be identified in the received light. In the present disclosure, an active depth sensing system that projects a distribution of light (e.g., structured light (SL), such as a distribution of light points, a flood light, and/or other shapes) is referred to as a SL system (with a SL projector).
Denser distributions of light (such as additional light points or more instances of focused light in an area than for sparser distributions of light) may result in a higher resolution of a depth map or a greater number of depths that may be determined. However, the intensity of individual light points are lower for denser distributions than for sparser distributions, and thus denser distributions of light may be more susceptible to interference from ambient light than sparser distributions of light. Thus, a sparser distribution may be more suitable for daylight scenes (with more interference), and a denser distribution may be more suitable for indoor or nighttime scenes (with less interference).
Many devices use a SL system in different types of lighting (with different amounts of interference). For example, a smartphone may include an active depth sensing system for face recognition, and the smartphone may be used indoors and outdoors. Many devices also include a flood illuminator. A flood illuminator may project a diffuse light onto a scene so that enough light exists in the scene for an image sensor to capture one or more images of the scene. In one example, a device (such as a smartphone) that performs face recognition may first determine if a face to be recognized (and/or identified) exists in the scene. In some implementations, the device may capture a two-dimensional (2D) image using flood illumination and then use the 2D image in conjunction with a three-dimensional (3D) image to recognize (and/or identify) a face (if any) in the image. Specifically, a light projector of the device may include a flood illuminator to project IR light onto a scene so that an IR sensor may capture the scene, and the device may determine from the capture if a face exists in the scene. If a face is determined to exist in the scene, the device may then use an active depth sensing system (e.g., via one or more light projectors) for face recognition and/or liveness confirmation. In some implementations, the device may use a proximity sensor to determine whether a face is present in the scene. In some aspects, a user of the device may determine when to turn on the light projector. In some other aspects, the device may be configured to turn on the projector automatically. In these ways, the device may use a 2D image (e.g., captured via flood illumination) in conjunction with a 3D image to enhance the performance of the device.
As discussed above, an active depth sensing system may use a single DFB laser to emit a single point of light through a coded DOE, which may reduce distortion and achieve higher-resolution images as compared with a VCSEL array. However, due to a damage threshold for the DFB laser, at continuous wave or long pulse (e.g., on the order of milliseconds) operation, the light emitted from the DFB laser may have a lower power (and thus be less bright) than the light emitted from the VCSEL array, and may therefore be more susceptible to ambient light (such as sunlight).
Rather than emitting light in a continuous wave long pulse light, aspects of the present disclosure describe a device including a DFB laser configured to output a sequence of relatively short bursts of light pulses (a “pulse burst”) that are each followed by a relatively long cooling period. Each pulse burst may include a small number of short light pulses. By repeating this pulse train burst and cooling cycle enough times, the device described herein may enable the single DFB laser to safely operate at a high enough power to illuminate a scene with brightness comparable to an array of VCSELs. Thus, SNR degradation from ambient light (i.e., noise) for the device may be reduced. In this way, aspects of the present disclosure may be used to enable, or otherwise improve, active depth sensing applications, such as, and not limited to: facial lock, facial recognition, facial modeling, animated emoji, user avatars, video conferencing, 3D modeling (such as automotive 3D modeling), 3D modeling for augmented reality (AR), gesture recognition, macro room imaging, among other appropriate applications. As one non-limiting example, aspects of the present disclosure may enable a mobile phone to capture a face of a user of the mobile phone (e.g., to enable the user to make a mobile payment) even while the user and the mobile phone are in bright conditions outdoors.
In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term “coupled” as used herein means connected directly to or connected through one or more intervening components or circuits. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring teachings of the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing the terms such as “accessing,” “receiving,” “sending,” “using,” “selecting,” “determining,” “normalizing,” “multiplying,” “averaging,” “monitoring,” “comparing,” “applying,” “updating,” “measuring,” “deriving,” “settling” or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
In the figures, a single block may be described as performing a function or functions; however, in actual practice, the function or functions performed by that block may be performed in a single component or across multiple components, and/or may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Also, the example devices may include components other than those shown, including well-known components such as a processor, memory and the like.
Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors, and are therefore not limited to specific devices.
The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system and so on). As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term “device” to describe various aspects of this disclosure, the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific implementations. For example, a system may be implemented on one or more printed circuit boards or other substrates, and may have movable or static components. While the below description and examples use the term “system” to describe various aspects of this disclosure, the term “system” is not limited to a specific configuration, type, or number of objects.
The SL system 100 may be used to generate depth information for a scene 106. For example, the scene 106 may include a face, and the SL system 100 may be used for identifying or authenticating the face. The SL system 100 may include a projector 102 and a receiver 108. The projector 102 may be referred to as a “transmitter,” “projector,” “emitter,” and so on, and should not be limited to a specific transmission component. Throughout the following disclosure, the terms emitter, projector and transmitter may be used interchangeably. The receiver 108 may be referred to as a “detector,” “sensor,” “sensing element,” “photodetector,” and so on, and should not be limited to a specific receiving component.
While the disclosure refers to the distribution as a light distribution, any suitable signals at other frequencies may be used (such as radio frequency waves, sound waves, etc.). Further, while the disclosure refers to the distribution as including a plurality of light points, the light may be focused into any suitable size and dimensions. For example, the light may be projected in lines, squares, or any other suitable dimension. In addition, the disclosure may refer to the distribution as a codeword distribution, where a defined portion of the distribution (such as a predefined patch of light points) is referred to as a codeword. If the distribution of the light points is known, the codewords of the distribution may be known. However, the distribution may be organized in any way, and the present disclosure should not be limited to a specific type of distribution or type of signal or pulse.
The projector 102 may be configured to project or emit a distribution 104 of light points onto the scene 106. The white circles in the distribution 104 may indicate where no light is projected for a possible point location, and the black circles in the distribution 104 may indicate where light is projected for a possible point location. In some example implementations, the projector 102 may include one or more light sources 124 (such as one or more lasers), a lens 126, and a light modulator 128. The projector 102 also may include an aperture 122 from which the emitted light escapes the projector 102. In some implementations, the projector 102 may further include a diffractive optical element (DOE) to diffract the emissions from one or more light sources 124 into additional emissions. In some aspects, the light modulator 128 (to adjust the intensity of the emission) may comprise a DOE. In projecting the distribution 104 of light points onto the scene 106, the projector 102 may emit one or more lasers from the light source 124 through the lens 126 (and/or through a DOE or light modulator 128) and onto the scene 106. The projector 102 may be positioned on the same reference plane as the receiver 108, and the projector 102 and the receiver 108 may be separated by a distance called the baseline (112).
In some example implementations, the light projected by the projector 102 may be IR light. IR light may include portions of the visible light spectrum and/or portions of the light spectrum that is not visible to the naked eye. In one example, IR light may include near infrared (NIR) light, which may or may not include light within the visible light spectrum, and/or IR light (such as far infrared (FIR) light) which is outside the visible light spectrum. The term IR light should not be limited to light having a specific wavelength in or near the wavelength range of IR light. Further, IR light is provided as an example emission from the projector 102. In the following description, other suitable wavelengths of light may be used. For example, light in portions of the visible light spectrum outside the IR light wavelength range or ultraviolet light. Alternatively, other signals with different wavelengths may be used, such as microwaves, radio frequency signals, and other suitable signals.
The scene 106 may include objects at different depths from the SL system (such as from the projector 102 and the receiver 108). For example, objects 106A and 106B in the scene 106 may be at different depths. The receiver 108 may be configured to receive, from the scene 106, reflections 110 of the emitted distribution 104 of light points. To receive the reflections 110, the receiver 108 may capture an image. When capturing the image, the receiver 108 may receive the reflections 110, as well as (i) other reflections of the distribution 104 of light points from other portions of the scene 106 at different depths and (ii) ambient light. Noise also may exist in the captured image.
In some example implementations, the receiver 108 may include a lens 130 to focus or direct the received light (including the reflections 110 from the objects 106A and 106B) on to the sensor 132 of the receiver 108. The receiver 108 also may include an aperture 120. Assuming for the example that only the reflections 110 are received, depths of the objects 106A and 106B may be determined based on the baseline 112, displacement and distortion of the light distribution 104 (such as in codewords) in the reflections 110, and intensities of the reflections 110. For example, the distance 134 along the sensor 132 from location 116 to the center 114 may be used in determining a depth of the object 106B in the scene 106. Similarly, the distance 136 along the sensor 132 from location 118 to the center 114 may be used in determining a depth of the object 106A in the scene 106. The distance along the sensor 132 may be measured in terms of number of pixels of the sensor 132 or a distance (such as millimeters).
In some example implementations, the sensor 132 may include an array of photodiodes (such as avalanche photodiodes) for capturing an image. To capture the image, each photodiode in the array may capture the light that hits the photodiode and may provide a value indicating the intensity of the light (a capture value). The image therefore may be the capture values provided by the array of photodiodes.
In addition, or in the alternative, the sensor 132 may include a complementary metal-oxide semiconductor (CMOS) sensor. To capture the image by a photosensitive CMOS sensor, each pixel of the sensor may capture the light that hits the pixel and may provide a value indicating the intensity of the light. In some example implementations, an array of photodiodes may be coupled to the CMOS sensor. In this manner, the electrical impulses generated by the array of photodiodes may trigger the corresponding pixels of the CMOS sensor to provide capture values.
The sensor 132 may include at least a number of pixels equal to the number of possible light points in the distribution 104. For example, the array of photodiodes or the CMOS sensor may include a number of photodiodes or a number of pixels, respectively, corresponding to the number of possible light points in the distribution 104. The sensor 132 logically may be divided into groups of pixels or photodiodes (such as 4×4 groups) that correspond to a size of a bit of a codeword. The group of pixels or photodiodes also may be referred to as a bit, and the portion of the captured image from a bit of the sensor 132 also may be referred to as a bit. In some example implementations, the sensor 132 may include the same number of bits as the distribution 104.
If the light source 124 emits IR light (such as NIR light at a wavelength of, e.g., 940 nm), the sensor 132 may be an IR sensor to receive the reflections of the NIR light. The sensor 132 also may be configured to capture an image using a flood illuminator (not shown for simplicity). As illustrated, the distance 134 (corresponding to the reflections 110 from the object 106B) is less than the distance 136 (corresponding to the reflections 110 from the object 106A). Using triangulation based on the baseline 112 and the distances 134 and 136, the differing depths of objects 106A and 106B in the scene 106 may be determined in generating depth information for the scene 106. Determining the depths may further include determining a displacement or a distortion of the distribution 104 in the reflections 110.
Although a number of separate components are illustrated in
For a light projector (such as the projector 102), the light source may be any suitable light source. In some example implementations, the light source 124 may include one or more distributed feedback (DFB) lasers. In some other example implementations, the light source 124 may include one or more vertical-cavity surface-emitting lasers (VCSELs).
A DOE is a material situated in the projection path of the light from the light source. The DOE may be configured to split a light point into multiple light points. For example, the material of the DOE may be a translucent or a transparent polymer with a known refractive index. The surface of the DOE may include peaks and valleys (varying the depth of the DOE) so that a light point splits into multiple light points when the light passes through the DOE. For example, the DOE may be configured to receive one or more lights points from one or more lasers and project an intended distribution with a greater number of light points than emitted by the one or more lasers. While the Figures may illustrate the depth of a DOE changing along only one axis of the DOE, the Figures are only to assist in describing aspects of the disclosure. The peaks and valleys of the surface of the DOE may be located at any portion of the surface of the DOE and cause any suitable change in the depth of portions of the DOE, and the present disclosure should not be limited to a specific surface configuration for a DOE.
If the light source 124 includes an array of lasers (such as a VCSEL array), a portion of the distribution of light points may be projected by the array. A DOE may be used to replicate the portion in projecting the distribution of light points. For example, the DOE may split the projection from the array into multiple instances, and a pattern of the projection may be a repetition of the projection from the array. In some example implementations, the DOE may be configured to repeat the projection vertically, horizontally, or at an angle between vertical and horizontal relative to the projection. The repeated instances may be overlapping, non-overlapping, or any suitable configuration. While the examples describe a DOE configured to split the projection from the array and stack the instances above and below one another, the present disclosure should not be limited to a specific type of DOE configuration and repetition of the projection.
In some implementations, the emitter 201 may be a distributed feedback (DFB) laser for emitting light pulses onto the scene. The DOE 205 may enable the emitter 201 to emit a distribution of light such as a known DOE dot pattern, a codeword DOE projection, a random dot projection or distribution, or the like. The diffuser 207 may transition the device 200 between one or more operating modes. For example, in a transmission mode, the diffuser 207 may be switched on to allow the emitter 201 to emit light pulses into a scene, and in a non-transmission mode, the diffuser 207 may be switched off to prevent the emitter 201 from emitting light pulses into the scene.
In some implementations, the sensor 202 may be a gated, global shutter (GS) sensor configured to receive reflected light pulses from the scene. In some aspects, the sensor 202 may be a composite CMOS image sensor. In some aspects, the sensor 202 may be based on a monolithic pixel array architecture, for example, with Time-Division Multiplexed Read (TDMR) capabilities.
In some implementations, the active depth controller 210 may be a computation element for calculating depth information. In some aspects, the active depth controller 210 may be configured to control (or otherwise operate) one or both of the emitter 201 and the sensor 202. In some aspects, the active depth controller 210 may be controlled, work in conjunction with, or otherwise be operated by one or more other components of the device 200, such as the processor 204 and/or the memory 206.
The device 200 may optionally include or be coupled to a display 214 and a number of input/output (I/O) components 216. The sensor 202 may be, or otherwise may be coupled to, a camera, such as a single camera, a dual camera module, or a module with any number of other camera sensors (not shown for simplicity). The signal processor 212 may be configured to process captures from the sensor 202. The device 200 may further include one or more optional sensors 220 (such as a gyroscope, magnetometer, inertial sensor, NIR sensor, and so on) coupled to the processor 204. The device 200 also may include a power supply 218, which may be coupled to or integrated into the device 200. The device 200 may include additional features or components not shown.
The memory 206 may be a non-transient or non-transitory computer readable medium storing computer-executable instructions 208 to perform all or a portion of one or more operations described in this disclosure. The processor 204 may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions 208) stored within the memory 206. In some aspects, the processor 204 may be one or more general purpose processors that execute instructions 208 to cause the device 200 to perform any number of functions or operations. In additional or alternative aspects, the processor 204 may include integrated circuits or other hardware to perform functions or operations without the use of software. While shown to be coupled to each other via the processor 204 in the example of
The display 214 may be any suitable display or screen allowing for user interaction and/or to present items (such as depth information or a preview image of the scene) for viewing by a user. In some aspects, the display 214 may be a touch-sensitive display. The I/O components 216 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and to provide output to the user. For example, the I/O components 216 may include (but are not limited to) a graphical user interface, keyboard, mouse, microphone and speakers, squeezable bezel or border of the device 200, physical buttons located on device 200, and so on. The display 214 and/or the I/O components 216 may provide a preview image or depth information for the scene to a user and/or receive a user input for adjusting one or more settings of the device 200 (such as adjusting an intensity of emissions by emitter 201, determining or switching one or more operating modes of the device 200, adjusting a field of emission of the emitter 201, and so on).
The active depth controller 210 also may include, or may otherwise be coupled to, a signal processor 212, which may be one or more processors to process captures from the sensor 202. The active depth controller 210 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.
The emitter 201 may vary its field of emission for different operating modes. In some example implementations, the emitter 201 may include a focusing apparatus for adjusting the size of the field of emission/transmission. In one example, mirrors attached to actuators (such as microelectromechanical systems (MEMS) actuators) may adjust a focus of the light emissions from the emitter 201. In another example, an adjustable holographic optical element (HOE) may adjust the focus of the light emissions from the emitter 201. In a further example, a formable diffractive optical element (DOE) (such as a piezoelectric material to adjust the shape) may be adjusted to focus the diffracted points of light emitted.
An example device 300 may be a smartphone, with an earpiece 310 and a microphone 312 for conducting phone calls or other wireless communications. A smartphone also may include a display 314 with or without a notch including the active depth sensing light projectors 302, illuminator 304, and the IR sensor 306. A flood illuminator 304 may project a diffuse IR light onto a scene for the IR sensor 306 to capture an image based on reflections of the diffuse IR light.
At time t0, the device may begin a frame exposure 401. During the frame exposure 401, the device may illuminate the scene using a sequence of two or more periods. As a non-limiting example, a first period is shown that starts at time to and ends at time t2, and a second period is shown that starts at time t2 and ends at time t4. Each of the two or more periods may include a transmission portion and a non-transmission portion. For purposes of discussion herein, each transmission portion may be referred to as a “pulse burst,” and each non-transmission portion may be referred to as a “cooling period.”
During each transmission portion of the sequence, such as from time t2 to time t3, the device may emit a plurality of light pulses onto the scene. Each of the light pulses may have a duration, which is shown as a time difference, tB−tA. In some implementations, the device may include one or more lasers (such as the light projector 302 of
The sequence includes a number of non-transmission portions between pulse bursts. As a non-limiting example, a first non-transmission portion is shown from time t1 to time t2. Each of the non-transmission portions may correspond to an absence of emitted light (such that the laser does not emit light pulses), for example, so that an operating temperature of the laser may decrease between pulse bursts. The device may prevent or disable the accumulation of photoelectric charge during each non-transmission portion, for example, by disabling the one or more sensors. In this manner, the device may prevent the one or more sensors from receiving ambient light (which may degrade SNR) during the non-transmission portions.
In some implementations, the one or more lasers may emit each of the light pulses at a selected transmit power level that exceeds a maximum specified power level for continuous or long pulse operation of the laser. As the one or more lasers emit the light pulses during the transmission portions, a temperature of the one or more lasers may approach or exceed a specified temperature value at which the one or more lasers may incur physical damage. A duration of each of the non-transmission portions may be selected to prevent damage to the one or more lasers, such as from the operating temperature of the one or more lasers being greater than the specified temperature value during a corresponding one of the transmission portions, by allowing the operating temperature of the one or more lasers to decrease to a level below the specified temperature value. In this manner, each of the non-transmission portions of the sequence may provide a cooling period for a preceding transmissions portion of the sequence. In some aspects, the duration of each non-transmission portion of the sequence may be an order of magnitude greater than the duration of each corresponding transmission portion of the sequence.
Since a gating duration for a traditional single DFB laser may be on the order of approximately tens of nanoseconds, a traditional GS sensor may not be capable of reliably turning on and off during emission that synchronizes with a given light pulse from the DFB laser. Aspects of the present disclosure may enable the GS sensor to be gated and synchronized with each pulse burst from the single DFB laser, for example, so that the device receives reflected light pulses during each transmission portion of the sequence and does not receive any or most of the light (for example, ambient light) during each non-transmission portion of the sequence. As one non-limiting example, the duration of each transmission portion may be approximately 1 μs, and the duration of each non-transmission portion may be approximately 10 μs. Each transmission portion may include approximately 25 light pulses, and the duration (or “pulse-width”) of each light pulse may be approximately 20 ns in an appropriate (relatively high) duty cycle, such as approximately 50%. In some aspects, the duty cycle also may adhere to a heat characteristic of the DFB laser. In some aspects, the light pulses may be emitted at a current of approximately 2-3 A. The DFB laser may be configured to emit a number of bursts (approximately 800-3200 in this example) sufficient to saturate the image. In some implementations, the number of pulses in each pulse burst and the number of pulse bursts may be dynamically adjusted based on system conditions, such as a measured SNR,
where μsig represents the laser signal, and where σbg represents the ambient light noise. SNR may be higher when the device is indoors than when the device is exposed to sunlight outdoors. Thus, in some implementations, the device may dynamically adjust at least one of a number of the transmission portions and a number of the plurality of light pulses based on environmental light conditions for the device. For example, the device may dynamically reduce the number of light pulses or the number of transmission portions when the device is indoors (or in other low-light conditions). As another example, the device may dynamically increase the number of light pulses or the number of transmission portions when the device is outdoors (or in other bright-light conditions).
As a non-limiting example, the sequence may include approximately 500-2000 periods, where each period includes a transmission portion and a non-transmission portion. In part by switching the GS sensor on at the beginning of each pulse burst (such as at time t0) and switching the GS sensor off at the end of each pulse burst (such as at time t1), aspects of the present disclosure may allow the single DFB laser to safely operate at a high enough power to illuminate a scene with brightness comparable to an array of VCSELs (e.g., approximately 2-3 W).
The sequence of periods for the frame exposure 401 is shown to include a number, n, of additional periods. Each of the n additional periods may include a corresponding transmission portion and a corresponding non-transmission portion (not shown for simplicity). During an entirety of the sequence (e.g., beginning at time t0), the device may continuously accumulate photoelectric charge indicative of the received light pulses. For example, each of the one or more sensors may be a photodiode configured to continuously receive photons from the received light pulses during each transmission portion of the sequence. The device may be configured to integrate the photoelectric charge to one or more storage nodes of the device, such as the memory 206 of
The sequence (and the frame exposure 401) may end after a final transmission portion (the “final pulse burst”) that starts at time tn and ends at time tn+1. In some implementations, the final transmission portion may not be followed by a corresponding non-transmission portion.
After the end of the sequence (e.g., at time tn+1), the device may transfer (or “extract”) the accumulated photoelectric charge to a readout circuit (not shown for simplicity). In some implementations, the device may readout the photoelectric charge in a manner similar to the GS pixel architecture disclosed in U.S. Pat. No. 9,332,200 B1. In some example implementations, the device may perform a number, m, of additional frame exposures after the frame exposure 401. The device may operate the same or similar to the frame exposure 401 during each of the m additional frame exposures.
The device receives, during each transmission portion, a plurality of light pulses reflected from the scene (702). In some implementations, a sensor may continuously receive photons during each transmission portion of the sequence, and may be prevented from receiving ambient light during each non-transmission portion of the sequence.
The device continuously accumulates photoelectric charge indicative of the received light pulses during an entirety of the sequence (703). In some implementations, the device may pause or disable the accumulation of photoelectric charge during each non-transmission portion of the sequence.
The device transfers the accumulated photoelectric charge to a readout circuit after an end of the sequence (704). In some implementations, the transferred accumulated photoelectric charge may be used to construct an image of the scene. In some aspects, the image may be a face of a user to be identified or authenticated by a mobile device such as a smartphone.
As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.
The various illustrative logics, logical blocks, modules, circuits and algorithm processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. The interchangeability of hardware and software has been described generally, in terms of functionality, and illustrated in the various illustrative components, blocks, modules, circuits and processes described above. Whether such functionality is implemented in hardware or software depends upon the particular application and design constraints imposed on the overall system.
The hardware and data processing apparatus used to implement the various illustrative logics, logical blocks, modules and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or, any conventional processor, controller, microcontroller, or state machine. A processor also may be implemented as a combination of computing devices such as, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry that is specific to a given function.
In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents thereof, or in any combination thereof. Implementations of the subject matter described in this specification also can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage media for execution by, or to control the operation of, data processing apparatus.
If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The processes of a method or algorithm disclosed herein may be implemented in a processor-executable software module which may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that can be enabled to transfer a computer program from one place to another. A storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection can be properly termed a computer-readable medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and instructions on a machine readable medium and computer-readable medium, which may be incorporated into a computer program product.
Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with this disclosure, the principles and the novel features disclosed herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7518759 | Hosier et al. | Apr 2009 | B2 |
| 7538307 | Lauxtermann | May 2009 | B1 |
| 8222586 | Lee | Jul 2012 | B2 |
| 8547458 | Iwasawa | Oct 2013 | B2 |
| 8638382 | Itonaga et al. | Jan 2014 | B2 |
| 9774801 | Hseih et al. | Sep 2017 | B2 |
| 9800807 | Hicks | Oct 2017 | B2 |
| 10129377 | Evans | Nov 2018 | B2 |
| 10466342 | Zhu | Nov 2019 | B1 |
| 20030058356 | Dicarlo et al. | Mar 2003 | A1 |
| 20050035381 | Holm et al. | Feb 2005 | A1 |
| 20050046715 | Lim et al. | Mar 2005 | A1 |
| 20050110884 | Altice, Jr. et al. | May 2005 | A1 |
| 20050110885 | Altice, Jr. et al. | May 2005 | A1 |
| 20050128327 | Bencuya et al. | Jun 2005 | A1 |
| 20070091190 | Iwabuchi et al. | Apr 2007 | A1 |
| 20080055441 | Altice | Mar 2008 | A1 |
| 20080083939 | Guidash | Apr 2008 | A1 |
| 20080170149 | Iida et al. | Jul 2008 | A1 |
| 20080309800 | Olsen et al. | Dec 2008 | A1 |
| 20090184349 | Dungan | Jul 2009 | A1 |
| 20090189234 | Mabuchi | Jul 2009 | A1 |
| 20100019130 | Lee | Jan 2010 | A1 |
| 20100060764 | Mccarten et al. | Mar 2010 | A1 |
| 20100276572 | Iwabuchi et al. | Nov 2010 | A1 |
| 20110101420 | Patel | May 2011 | A1 |
| 20110163223 | Guidash | Jul 2011 | A1 |
| 20110188054 | Petronius et al. | Aug 2011 | A1 |
| 20120050597 | Maeda et al. | Mar 2012 | A1 |
| 20120274744 | Wan et al. | Nov 2012 | A1 |
| 20130264467 | Hong et al. | Oct 2013 | A1 |
| 20130307103 | Lin et al. | Nov 2013 | A1 |
| 20130334403 | Kozlowski | Dec 2013 | A1 |
| 20140008520 | Raynor | Jan 2014 | A1 |
| 20140077057 | Chao et al. | Mar 2014 | A1 |
| 20140263959 | Hsu et al. | Sep 2014 | A1 |
| 20140320718 | Fan | Oct 2014 | A1 |
| 20150229861 | Yakemoto | Aug 2015 | A1 |
| 20150236065 | Eom | Aug 2015 | A1 |
| 20150271476 | Wan | Sep 2015 | A1 |
| 20150373322 | Goma et al. | Dec 2015 | A1 |
| 20160360074 | Winer et al. | Dec 2016 | A1 |
| 20180164440 | Fried et al. | Jun 2018 | A1 |
| 20190072771 | Hall | Mar 2019 | A1 |
| 20190181171 | Tadmor | Jun 2019 | A1 |
| 20190346537 | Krelboim | Nov 2019 | A1 |
| 20200154015 | Lyu | May 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2482402 | Sep 2011 | CA |
| 1233806 | Nov 1999 | CN |
| 101188245 | May 2008 | CN |
| 101753866 | Jun 2010 | CN |
| 101821850 | Sep 2010 | CN |
| 1953193 | Dec 2011 | CN |
| 103367380 | Oct 2013 | CN |
| 1493017 | Jul 2012 | EP |
| 2000165754 | Jun 2000 | JP |
| 2007013089 | Jan 2007 | JP |
| 2010165854 | Jul 2010 | JP |
| 2012502469 | Jan 2012 | JP |
| 19990083635 | Nov 1999 | KR |
| 20070043656 | Apr 2007 | KR |
| WO-2009051379 | Apr 2009 | WO |
| WO-2014051306 | Apr 2014 | WO |
| WO-2014064837 | May 2014 | WO |
| Entry |
|---|
| Hondongwa D., et al., “Quanta Image Sensor (QIS): Early Research Progress,” In Proc. 2013 Optical Society of America (OSA) Mtg. on Imaging Systems and Applications, Arlington, VA, Jun. 24-27, 2013, pp. 1-3. |
| Vogelsang T., et al., “Overcoming the Full Well Capacity Limit: High Dynamic Range Imaging Using Multi-Bit Temporal Oversampling and Conditional Reset,” Rambus Inc., International Image Sensor Workshop, 2013, pp. 1-4. |
| International Search Report and Written Opinion—PCT/US2020/047644—ISA/EPO—dated Feb. 2, 2021. |
| Number | Date | Country | |
|---|---|---|---|
| 20210067662 A1 | Mar 2021 | US |
