Patent Grant
10962628
The described embodiments relate in general to light detectors, such as may be used, for example, as part of a scanning light sensor or a scanning depth sensor (SDS) system. In SDS systems, objects in a field of view (FOV) can be detected by emitting light into the FOV and receiving reflections of the emitted light, such as on a photodetector array of light sensing pixels. Particular embodiments described include light detectors in which the light sensing pixels include single-photon avalanche diodes (SPADs). Such light detectors are termed SPAD detectors, and the pixels that include a SPAD diode are termed SPAD pixels. More particularly, the present embodiments relate to various spatial temporal weighting schemes that can be used with SPAD detectors or other light detectors.
Scanning light sensors and scanning depth sensor (SDS) systems can be used as stand-alone electronic devices, or as components of other electronic devices, such as digital cameras and 3D imagers. An SDS typically operates by emitting light into a field of view (FOV) and detecting reflections of the emitted light. By measuring the time between emission of the light and reception of its reflection, a distance to an object can be calculated. In some SDSs the light is emitted in one or more narrow beams, such as by lasers. In line scan implementations, an emitted beam is swept or scanned across the FOV and the reflected light beam is received at respective specific locations on an array of light sensing pixels. As the emitted beam is scanned over the FOV, the location of the received reflected beam sweeps across the array. The light detector that detects the reflected light beams in an SDS may be a SPAD detector, in which the array of light sensing pixels uses SPADs for enhanced light detection capabilities.
SPAD detectors are utilized in a variety of applications, such as low-light detection applications, time-of-flight (TOF) applications, and time-correlated single-photon counting applications. A SPAD detector typically includes an array of light sensing pixels, with each pixel including a SPAD. Each SPAD is a photosensitive region that is configured to detect low levels of light (down to a single photon) and to signal the arrival times of the light. An individual SPAD includes a diode reverse biased into its avalanche region. When a photon impinging on a SPAD of a SPAD pixel is part of a reflection from an object of an emitted light pulse, the output signal can be used to estimate the arrival time of the photon at the SPAD after emission of the light pulse. Multiple such arrival times can be used to estimate a distance to the object with greater certainty.
A SPAD typically has a constant sensitivity to photons, regardless of the distance between the SPAD and an object from which a photon has been reflected. In situations in which the object is close to the SPAD detector, the photons in the light reflected from that object impinge on the SPADs of the SPAD detector sooner and often in higher numbers than in situations in which the object is farther from the SPAD detector. Additionally, the photons in light reflected from an object having a higher reflectivity may impinge on the SPADs in higher numbers than photons in light reflected from an object having a lower reflectivity. When one or more photons impinge on a SPAD, the photons may trigger an avalanche event. A triggered SPAD (i.e., a SPAD in which an avalanche event has been triggered) will produce an output pulse signal corresponding to the SPAD's trigger time. Following an avalanche event or trigger, a SPAD will be saturated and have a recharging “dead time” during which the SPAD cannot be used to detect photons. Emitted photons reflected from a closer or more reflective object can saturate a SPAD at an early time and cause the SPAD to be unusable for detecting photons reflected from objects within a desired detection range. This can result in incorrect or unavailable estimates of the distance to an object within an intended detection range.
A TOF is a measurement of the amount of time that elapses between an emitter emitting a light pulse toward an object and the SPAD detector detecting photons in a light pulse that reflects off the object. In general, the light that reflects off an object that is close to the SPAD detector has a shorter TOF. The photons in the reflected light can impinge on the SPAD detector more quickly and in higher numbers than when an object is farther from the SPAD detector. The photons in the light reflected from the closer or more reflective object can saturate the SPADs more quickly, which can result in an incorrect estimate of the TOF.
Conversely, the light that reflects off an object that is farther from the SPAD detector has a longer TOF, and a lower intensity reflected light pulse. During the longer TOF, the amount of ambient light that is measured by the SPAD detector can overwhelm the reflected light pulse that was originally emitted by the emitter. It can be challenging to discern the reflected light pulse from the measured ambient light, which can make it difficult to correctly estimate the TOF.
Methods and devices are presented that, in some embodiments, address the problem of saturating SPADs and/or of poor signal to noise ratio (SNR).
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In a first embodiment, a method of estimating a distance of an object from a sensor includes emitting a sequence of light pulses, with one emitted light pulse emitted in an initial segment of a sequence of pulse repetition intervals (PRIs). The method includes receiving, during each of a plurality of the PRIs, a received light pulse at a light sensing pixel of the sensor, and determining a time-of-flight of each received light pulse. Each time-of-flight is determined from a discrete subinterval of time of the PRI during which the received light pulse was received. For each time-of-flight, a respective weight is determined based on the time-of-flight and the position within the sequence of PRIs of the PRI during which the received light pulse corresponding to the time-of-flight was received. The determined weight is used to form a weighted count of the times-of-flight. A distance to the object is estimated based on the weighted count of times-of-flight.
In additional and/or alternative embodiments, the method includes receiving a determination that the object is located at a far distance in a field of view, and consequently using weights for the times-of-flight of the received light pulses that are larger for received light pulses having higher signal to noise ratio than for received light pulses with lower signal to noise ratio. Such light pulses having higher signal to noise ratio may be received during a middle section of the sequence of PRIs. A spatial matched filter may be used to form the weighted count of the times-of-flight.
In additional and/or alternative embodiments, the method includes receiving a determination that the object is located at a near distance in a field of view, and consequently using weights for the times-of-flight of the received light pulses that are larger for received light pulses having reduced intensity than for received light pulses without reduced intensity. Such light pulses may be received during a peripheral section of the sequence of PRIs.
In a second embodiment, a method of operating a light sensor includes emitting a sequence of outgoing light pulses over a sequence of time intervals, with one outgoing light pulse being emitted during an initial segment of each time interval of the sequence of time intervals. The method includes receiving a first set of received light pulses during an initial section of the sequence of time intervals, and detecting whether the object is present at a near distance within a field of view of the sensor. The detection is based on an aggregation of times-of-flight of received light pulses in the first set. In the first case that an object is not detected in the near distance, the method includes receiving a second set of received light pulses during time intervals that follow the initial section of the sequence of time intervals. In this first case the method includes forming a first weighted count of times-of-flight of the received light pulses in the second set of received light pulses using a first set of weights, and determining a distance to the object based on the first weighted count. In the second case that an object is detected in the near distance, the method includes receiving a third set of received light pulses during the time intervals that follow the initial section of the sequence of time intervals. In this second case the method includes forming a second weighted count of times-of-flight of the third set of received light pulses and determining a distance to the object using the second weighted count. The weights of the first weights are larger for received light pulses with higher signal to noise ratio. The weights of the second set of weights are larger for received light pulses with reduced intensity.
In further embodiments, the weights of the first set of weights are larger for times-of-flight of received light pulses that were received in a middle section of the sequence of time intervals than for times-of-flight of received light pulses that are received in peripheral sections of the sequence of time intervals. The weights of the second set of weights are larger for times-of-flight of received light pulses that are received in a peripheral section of the sequence of time intervals than for times-of-flight of received light pulses that are received in a middle section of the sequence of time intervals.
In further embodiments, the light sensor includes a photodetector array including single-photon avalanche diode pixels.
In a third family of embodiments, an electronic device is disclosed that includes a scanning light sensor system that includes an emitter, a detector that includes an array of light sensing pixels, and a processing device. The processing device is linked to the emitter and the detector, and is configured to cause the emitter to emit an emitted light pulse during an initial segment of each pulse repetition interval (PRI) of a sequence of PRIs. The detector is configured to receive, during each of a plurality of PRIs of the sequence of PRIs, a received light pulse and to determine a time-of-flight for that light pulse. The time-of-flight is determined from a discrete subinterval of time of the PRI in which the received light pulse was received. The processing device is also configured to determine a weight for each time-of-flight, each weight determined using at least a position within the sequence of PRIs of the PRI during which the received light pulse was received. The weights are used to form a weighted count of the times-of-flight using the weights. The weighted count is used to estimate a distance to an object based on the weighted count of the times-of-flight.
In further embodiments, the weights are determined using the time-of-flight and the position within the sequence of the PRIs with a look-up table (LUT) having a first axis representing the position within the sequence of PRIs, and a second axis corresponding to the discrete subintervals of time.
The disclosure will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements.
The use of cross-hatching or shading in the accompanying figures is generally provided to clarify the boundaries between adjacent elements and also to facilitate legibility of the figures. Accordingly, neither the presence nor the absence of cross-hatching or shading conveys or indicates any preference or requirement for particular materials, material properties, element proportions, element dimensions, commonalities of similarly illustrated elements, or any other characteristic, attribute, or property for any element illustrated in the accompanying figures.
Additionally, it should be understood that the proportions and dimensions (either relative or absolute) of the various features and elements (and collections and groupings thereof) and the boundaries, separations, and positional relationships presented therebetween, are provided in the accompanying figures merely to facilitate an understanding of the various embodiments described herein and, accordingly, may not necessarily be presented or illustrated to scale, and are not intended to indicate any preference or requirement for an illustrated embodiment to the exclusion of embodiments described with reference thereto.
Reference will now be made in detail to representative embodiments illustrated in the accompanying drawings. It should be understood that the following descriptions are not intended to limit the embodiments to one preferred embodiment. To the contrary, it is intended to cover alternatives, modifications, and equivalents as can be included within the spirit and scope of the described embodiments as defined by the appended claims.
The following disclosure relates generally to scanning light sensors and scanning depth sensor (SDS) systems (such as light detection and ranging (LIDAR) sensor systems). Such systems and sensors may use SPAD detectors (i.e., light detectors that use single-photon avalanche diodes (SPADs)). Such scanning light sensors and SDS systems operate by emitting light into a field of view (FOV) and observing reflections of the emitted light to detect whether there is an object in a field of view. In some embodiments, SDS systems use a line scan system in which the field of view (FOV) is swept with a beam of light pulses. Based on the time between emission of the light and the corresponding observed reflections on a light detector it is possible to estimate a distance to the object. Such systems and sensors may have as their light detector a photodetector array of light sensing pixels.
Embodiments may have a single-photon avalanche diode (SPAD) detector as its light detector. The SPAD detector includes an array of light sensing SPAD pixels (hereinafter, just “pixels”), wherein each such pixel includes a SPAD. Each pixel may also include associated biasing and/or control circuit elements, such as one or more of a quenching transistor and/or a gating transistor operably connected to the SPAD. Further circuit elements are described below for various embodiments. Times of flight (TOF) between the emitted pulses and received reflections of the emitted light pulses are used to calculate a distance to the object. Various spatial temporal weighting schemes that can be used with the SPAD detectors, and with other light detectors, are disclosed. Various techniques result in a histogram having bins that have each been incremented by differently weighted counts of detected photons.
Embodiments are discussed below with reference to
A processing device 108 is operably connected to the emitter 102 and to the detector 104. The processing device 108 causes the emitter 102 to emit light towards the object 106 (emitted light represented by arrow 110). The light reflected from the object 106 and/or the scene may be detected by the detector 104 (reflected light represented by arrow 112). The processing device 108 receives the output signals from the detector 104 or receives conditioned output signals from intervening signal processing components (not shown). The processing device 108 processes the output signals to determine one or more characteristics associated with the reflected light, the object 106, and/or the scene. The particular components and operations discussed for system 100 are exemplary; in other embodiments the operations discussed may be distributed among the components differently.
The system 100 may be used as part of an electronic device, such as a camera, in a mobile phone, that scans a field of view (FOV). In general, in scanning systems, light is emitted into the FOV and information about objects or targets in the FOV is determined from reflections of the emitted light. Scanning systems may emit light in multiple directions from multiple emitters, or by sweeping a light (such as from a laser) in one or more directions across some or all of the FOV. A scanning system may use multiple sweeping light emitters, which may sweep the FOV in different directions or patterns.
Alternatively, the system 100 may be part of an electronic device in which the illumination of the FOV is not scanned but rather is illuminated in fixed directions, such as by one or multiple emitters. In such systems (e.g., fixed pattern systems), one or multiple light pulses may be emitted (e.g., multiple contemporaneously-emitted pulses), and each emission may be directed or disbursed in one or more directions. For example, in a facial recognition system, multiple directions may be selected for a first set of simultaneous emissions. The various reflected pulses may then be used to detect the distinguishing facial features of a user. For a second set of emissions, the directions may be reselected and varied. In some of the embodiments disclosed herein, the reflected pulses may be directed (such as by a lens) to traverse the light sensing pixels (e.g., SPAD pixels) in the light detector.
In some embodiments, a SPAD detector is used in a line scan system.
Collectively, the emitted light pulses 218 are referred to herein as an emitted light beam, or just a light beam 210. The light beam 210 may have an intended illumination area, volume, or emission pattern at any given moment in time. The light beam 210 is steered or directed towards the FOV 212 so that only a section 214 (e.g., a line) of the FOV 212 is illuminated at a time. The FOV 212 is scanned section-by-section during a FOV detection period. The FOV detection period is the time period needed to scan a selected part of the FOV.
The light that is returned to the device (e.g., via reflections off an object and/or the scene in the FOV 212) is received by a lens 216 that directs the light onto the SPAD detector 206. Since the light beam 210 is a series of light pulses, the reflected light is comprised of a series of light pulses. As will be described in more detail later, sections of the pixels in the SPAD detector 206 detect the reflected light pulses through a series of line scan operations. Each scan operation involves the emission of multiple light pulses and detection of reflected photons by selected pixels of the array. Each line scan operation scans or reads out the pixels in a section of the pixel array (e.g., two or three columns at a time). Reading out pixels can involve receiving the output signals produced by the pixels' SPADs, and possibly performing amplification or other conditioning of the output signals. When the line scan operation for one section of pixels is complete, another section of pixels is scanned. In one embodiment, the next section of pixels includes some of the pixels in the previous line scan operation. In another embodiment, the next section of pixels includes different pixels from the pixels in the previous line scan operation. This process repeats until all of the pixels within a chosen subset of pixels of the array have been scanned.
In one embodiment, a beam-steering element 208 (e.g., a mirror) is positioned in the optical path of the emitter 204 to steer the light beam 210 emitted by the emitter 204 towards the FOV 212. The beam-steering element 208 is configured to control the propagation angle and path of the light beam 210 so that only a section 214 of the FOV 212 is illuminated at a time.
The light beam 210 can be generated and/or steered differently in other embodiments. For example, the emitter 204 can include multiple emitters that each emits light toward a different section of the FOV 212.
An electronic timing control system (such as part of processing device 108) can deactivate some or all of the pixels in the array of light sensing pixels (hereinafter “pixel array” or just “array”) during emission of each pulse of light to preclude the pixels from being saturated or giving a false signal. In some embodiments the electronic timing control system can then send a set of timing control signals to the emitter 204 to initiate or control emission of the sequence of light pulses. The electronic timing control system may subsequently send an activation signal to selected pixels in the array during times when no light is being emitted so that the activated pixels become configured to detect reflections of the emitted light pulses. The electronic timing control system can thus coordinate a particular activated pixel with a sequence of emitted or outgoing light pulses whose reflected pulses are expected to impinge on the activated pixel.
In an individual pixel of a SPAD detector, a SPAD is activated by being placed, such as by accompanying transistors, into a reversed biased state. In particular, a SPAD is operated in the avalanche region of reverse bias. When one or more photons enter the SPAD, charge carriers are created that migrate to an electrode. In so doing, they cause a cascade that increases the number of charge carriers leading to a measurable current spike. Surrounding circuitry, also called an analog front-end (AF), can amplify the current spike and transmit a signal indicating the reception of the photon(s). To save energy and prevent false positive reception signals, the SPAD can be deactivated (e.g., biased away from a reverse breakdown region) when its light detection operations are not expected or desired.
As mentioned previously, while the SPAD detector 302 may use a SPAD technology, the embodiments disclosed herein may use photodetector arrays based on other technologies, such as NMOS, PMOS, or CMOS photodetector arrays.
The emitter 300 may be a laser or other suitable light source that emits light 308 towards an object or FOV over a given period of time. In some embodiments, such as in a line scan system, the emitter 300 repeatedly emits a light pulse (e.g., light pulses 218 in
Various factors interfere with the ideal detection and distance measuring scheme just described. One problem is that ambient light in the FOV may be received and erroneously treated as a reflection of an emitted light pulse. Some sensor systems may be configured so that the emitter sends outgoing light pulses having a narrow frequency band, and the cover layer 306 configured to filter all light except in that narrow frequency band. Even so, there still may be received ambient light photons that produce output signals. Some embodiments deal with this problem by receiving multiple reflected light pulses over a large number of emitted light pulses and detecting a spike at a calculated distance value above a background level of calculated distances produced by ambient light signals.
Further, not all emitted or outgoing light pulses may result in a reflected pulse that is detected or detectable, such as due to absorption at the object or in the intervening air, slight misdirection of the reflected light, errors in the electronics, or other factors. As described further below, even when a reflected light pulse is detected, a first reception problem is that the number of reflected light pulses detected from an object far from the emitter may be on the order of the number of output signals produced by photons from ambient light. Thus there could be no statistically detectable spike in the TOFs above those produced by the ambient light. A second reception problem is that, for an object near the emitter, the received reflected light pulses may saturate the SPADs of the pixels early in the durations of the light pulses and induce a bias in the peak of the calculated TOF. This second reception problem will be explained further below in relation to
The top line of the top graph of
The second line of the top graph of
The third line of the top graph in
During the third PRI, a reflected light pulse 416 is detected. In the case that the object is within a maximum detection distance, the reflected light pulse 416 is a reflection of the emitted light pulse 410B. The TOF 414A is obtained by the TDC, as explained below. During the fifth shown PRI, another reflected light pulse 420 is detected, which is a reflection of emitted pulse 410D and which has a respective TOF 418A.
The bottom plot in
The vertical axis represents the counted number of reflected light pulses in each of the N bins of the histogram, each bin corresponding to a respective discrete subinterval of time of a PRI. The TDC can comprise a timer and circuitry to rapidly address N memory locations and increment the number in the memory location for efficient and fast counting of the received light pulses.
As will be clear to one of skill in the art, a histogram is a visual display of the counts of the TOF values, and as used hereinafter “histogram” will refer to any counting of the TOF values, such as in electronic memory locations, and need not be a visual display.
Over a large number of PRIs, a number of light pulses may be detected that do not arise as reflections of emitted light pulses, but from background ambient light or other falsely triggered pulses in, for example, a SPAD pixel. “Received light pulses” will refer to either the actual reflections of emitted light pulses or such erroneous detections of ambient light. Even detections of actual reflected emitted light pulses may show statistical variation in the discrete subinterval during which they are detected. This is indicated by the first TOF 414A being counted in the bin 414B, and the second TOF 418A being counted in the bin 418B. Over the large number of PRIs, the statistical variation of the TOFs of the reflected light pulses may produce a peak 422 in the histogram that is above the background noise level 424 of detected light pulses not arising as reflections of emitted light pulses. The discrete subinterval corresponding to the peak 422 can then be taken as the TOF and used to obtain the distance to the object.
The method just described counts each signal of a received light pulse equally. Embodiments to be described below apply a weighted counting.
Even with such a statistical calculation, there are still at least the two challenges previously mentioned for objects far and near in the field of view. For far objects, the reflected light pulses would be detected during discrete subintervals of the PRIs that are toward the right end of the bottom graph in
Referring again to
The result may be that photons received at time 432 and thereafter do not cause a current or output from the SPAD pixel. As a result, the counts of arrival times would be biased toward the early arriving photons in the light pulse, and so have the skewed and narrower graph 434, which has a peak magnitude at time 436. As a result, there is a bias or offset between the true peak time 432 of the intensity waveform 430 of the received reflected light pulse and the time 436 of the peak of the arrival times inferred from the narrower graph 434. For example, in some embodiments the received intensity waveform 430 has a width (such as a full width at half maximum, FWHM) of approximately 2 ns, and the bias 438 is approximately 1 ns. This bias in a TOF can result in an error of approximately 15 cm for the distance. This may be unacceptable for some situations, such as for autofocusing on a digital camera.
The operations discussed in relation to
For a scanning depth sensor sweeping the FOV, the beam of reflected light pulses may stepwise move from right to left, as indicated by the arrow 506, with the beam sweeping vertically in each step. When the traversal pattern of the beam is known, only those light sensing pixels in the anticipated location of the beam need to be activated for reception of the reflected light pulses. This can allow for a reduction in power use by the photodetector array 500 but requires timing and location determination of the paths of the reflected beam across the photodetector array 500.
In some embodiments, approximate determination of the time and location of the path 502 of the beam on the photodetector array 500 can be provided by processing that occurs off the photodetector array 500. For example, when the scanning depth sensor is used with a line scan system as in
Depending on the distance of the object in the FOV from the scanning sensor, either output signals from received reflected light pulses that are in PRIs of the peripheral sections 602 and 604, or those from PRIs in the middle section 606, may yield more accurate TOF values for building a histogram. The difference in accuracy may result from the distance of an object in the FOV.
The received reflected light pulses of the middle section 606 of the PRIs arrive at the pixel with a maximum of expected intensity and so will have higher signal to noise ratio than received reflected light pulses in the peripheral sections 602 and 604. When the object is either of middle or far distance in the FOV, such received reflected light pulses could be expected to yield better TOF values, and so should be emphasized in a counting of the TOF values. However, when the object is close within the FOV, the bias problem in the TOF values discussed above in relation to
At stage 702 a pixel is activated to begin detection of received light pulses. A processing device may have determined that reflections of emitted light pulses are beginning or about to begin overlapping the pixel. A histogram is also activated to begin receiving counts of TOF values corresponding to received light pulses. The histogram can be implemented as a block of memory locations that each store and update a count value. The number of memory locations can equal a number of discrete subintervals of time of the PRI, as explained below. The process described as follows for one PRI is repeated for all PRIs in a sequence of PRIs, the sequence of PRIs being those for which a reflected light pulse from an object can be expected to impinge on the pixel. This coordination of pixel activation with expected reflected light pulses may be accomplished by a processing device, such as processing device 108. In the illustrative embodiment discussed in
In stage 704, during one PRI, the pixel is active and may detect photons, either from an actual reflection of the emitted light pulse of that PRI, or from another source, such as an ambient light source. In the event of reception of received light pulse or signal, the pixel can produce an output signal that can be received by an analog front end (AF) circuit and conditioned before being sent to subsequent circuitry. Such signal conditioning can include amplifying by transistors and buffering.
At stage 706, a time-to-digital converter (TDC) circuit determines a time-of-flight. Recall that a TDC circuit may operate by cyclically counting the number of completed discrete subintervals within each PRI. The TDC can be triggered to stop counting upon reception of the output signal from the AF circuit. The number of completed discrete subintervals, multiplied by the length of time of the discrete subintervals, then is taken as the TOF for the received light pulse. When there are N many discrete subintervals of equal duration, the duration is the period of the PRI/N. If the discrete subintervals used do not have equal duration, the TOF is taken as the sum of durations of the previous discrete time subintervals. In some embodiments, the TDC could then trigger a counter to update the count value in memory location of the histogram. However, just updating a count by 1, as can be used in some embodiments in certain conditions, would not account for whether the object is near or far, nor for the intensity of the received light pulse.
At stage 708, instead of updating a bin of the histogram count by 1, a weight is selected and the histogram will store a weighted count. The weight is a number which may be different from 1 and which reflects how much to emphasize a TOF value when updating the histogram. Weights may be real-valued or integer valued, depending on data types used in the histogram. For example, if it is determined that the weight should be 2, reflecting a strong emphasis for the TOF value, the histogram location corresponding to the TOF is updated by 2, rather than 1. In another example, if instead the weight is determined to be 0.6, reflecting less confidence that the received light pulse that produced the TOF actually arose as a reflection from an object, the corresponding location in the histogram would only be updated with 0.6. In some embodiments, the weight is selected based on information about the position of the PRI in the sequence of PRI, and/or on the TOF value, as explained below.
At stage 710 the histogram is updated based on the TOF and the determined weight. The circuitry may be formed so that the memory address of the histogram corresponding to the TOF can be quickly accessed and updated.
At stage 712, the method either repeats if there are subsequent PRIs in the sequence of PRIs for the pixel, or else terminates and proceeds to the next stage. At termination the completed histogram contains weighted counts corresponding to each TOF.
At stage 714, from a completed histogram an estimate is made for a distance to an object in the FOV. In some situations, such as if no TOF value in the histogram has a count that exceeds a threshold, the decision may be made that no object is detected in the FOV.
At stage 802 a decision is made from received information about whether an object is at a far range within the FOV or that there is a high level of ambient light. If neither condition holds, then at stage 804 higher weight values are applied to TOF values arising from received light pulses detected at the pixel during PRIs from the peripheral sections (i.e., not in the middle section of PRIs in the sequence of PRIs) 602 and 604. Those TOF values will more likely not have the bias in the TOF discussed above. As in this case the ambient light is also low, a TOF arising from a low intensity received light pulse at the pixel is more likely to be from an actual reflected pulse from an object. Consequently, it can be emphasized more in the weighted count; for example, by incrementing the histogram location for that TOF by a weight greater than 1.
In the case of stage 804, a TOF value produced from a received light pulse during a PRI within the middle section 606 may be more likely to have a bias in its reception time, i.e., its TOF. Such TOF values may either be discarded entirely (i.e., a weight of 0 is assigned to it), or it can be counted in the histogram with a weight less than 1.
If, instead, at stage 802 it is determined that the object is at a far range in the FOV, or that ambient light is high, then those TOF values produced from light pulses with reduced intensity detected during a PRI of the peripheral sections 602 and 604 are more likely to be erroneous values. In this case flow passes to stage 806.
At stage 806, a TOF value arising during PRIs from the middle section 606 would be more likely to have arisen as an actual reflection from an object of the emitted pulse of that PRI. So a larger weight is selected for updating the count in the histogram for that TOF.
From both stages 806 and 804, flow passes to stage 808 the histogram is updated using the determined weight. Flow may then proceed to stage 712 of the method 700 of
In these methods the weight selected for a TOF during a particular PRI uses at least the position of that PRI within the sequence of PRIs. At least two lists of weights may be used: a first for the weights to be used in stage 804 and a second for the weights to be used in stage 806.
The horizontal axis gives the TDC histogram bin number that results when a received light pulse is detected during the corresponding discrete subinterval of time of the PRI.
The shown four levels of shading 902 in the LUT 900 indicate the weight value corresponding to each index pair, an index pair being one TDC histogram bin number and one PRI number. While the example 900 only depicts four weights for simplicity, one of skill in the art would recognize that more or fewer weights can be used in such a LUT, that different weight values may be used, that there may be more or fewer than 20 TDC histogram bin numbers, and that more or fewer than 6000 PRIs within a sequence of PRIs for a pixel.
The columns on the right side of the LUT correspond to received light pulses detected in one of the late discrete subintervals within the PRI, i.e., having a long time-of-flight (as a fraction of the PRI). As such, a column from the right side of the LUT can be used as the weights for stage 806. The weights as listed down the rightmost column of the LUT 900 increase from low weights to large weights in the middle section of PRIs, and then decrease back to low weights. Thus the weights can emphasize or weight higher count values to the histogram for received light pulses detected during the middle section, and decrease or weight lower the count values for received light pulses from the peripheral sections of the sequence of PRIs.
Analogously columns toward the left or center of the LUT correspond to received light pulses detected during one of the early discrete subintervals within the PRI, and so would indicate a reflected pulse having short or moderate time-of-flight. The weights as listed down the leftmost column of the LUT 900 decrease from large weights to low weights in the middle section of PRIs, and then increase back to large weights. Thus the weights can emphasize or increase count values to the histogram for received light pulses detected during the peripheral sections of the sequence of PRIs, and decrease the count values for received light pulses from the middle section of the sequence of PRIs.
At stage 1002 the position of the PRI is received. The position may be represented as an integer or fraction of the full length of the sequence of PRIs.
At stage 1004, the pixel is active to detect received light pulses or photons during the PRI. If no light pulse is detected, no TOF is created by the TDC circuit to add to the histogram.
At stage 1006, when a light pulse is detected at the pixel, the pixel's output is used by the AF circuitry to generate an output signal. The TDC circuitry can creates a value representing the discrete subinterval of the PRI in which the light pulse was detected. The value may be: an integer that counted the number of discrete subintervals of the PRI until the light pulse was received, an address or histogram bin number for the location in the histogram memory in which the count for the time-of-flight is updated and maintained, or another value that correlates with the time-of-flight. For explanatory purposes only, it will assumed that the TDC value is a histogram bin number.
At stage 1008, the position of the PRI and the TDC histogram bin number are used as indices to locate a weight in the LUT. The weights in the LUT may be of a data type compatible with the data types of the histogram.
At stage 1010 the histogram is incremented, in the location given by the TDC histogram bin number, by the value of the weight, or by the weight multiplied by another number.
The order in which the stages of
An advantage of the method of
In additional and/or alternative embodiments to be described now, rather than obtaining preliminary knowledge of the distance to an object by external means, the sequence of PRIs itself can be used to obtain a preliminary detection of an object within the FOV.
The light sensing pixels of the sensor may be triggered by received light signals or pulses that are not actual reflections of the outgoing light signals or pulses. The sequence of time intervals for which reflections are expected at the pixel may be sufficiently numerous to permit a statistical detection of TOFs of actual reflections of the outgoing light signals above a background level that results from light pulses originating from ambient or other spurious sources. The detected TOFs can then be used to calculate a distance to the object in the FOV.
The methods of
At stage 1102, detections are made of TOFs of received light pulses during PRIs from an initial set of the PRIs for which a reflected light pulse could impinge on the particular pixel. The TOFs may be determined by a TDC circuit and its counts of discrete subintervals of time within each time interval. The TOFs may be weighted as described below, and then the weighted counts used to update a histogram. In the particular example of
Because the PRIs in the initial set are from the beginning of the whole sequence of PRIs for the pixel, the reflected light pulses within those PRIs would likely only partially impinge on the pixel, and so are likely to be of low intensity. For example, they may be from the peripheral section 602 of the PRIs as shown in
At stage 1104, the aggregated counts of TOFs are used to determine if an object is in the FOV, and if so, in some cases determine an initial estimate for the distance to the object. The decision as to whether an object likely is in the FOV can be made on the basis of the histogram made using just the aggregated counts of TOFs calculated during the initial set of PRIs. As shown in
At stage 1106 a positive determination has been made that an object likely is in the FOV. Since the positive determination was made based on TOFs from time intervals in a leading peripheral section of the sequence of time intervals and the weights used to form the aggregated counts emphasized TOFs that would arise from an object near in the FOV, such a positive determination is more likely when there actually is both an object in the FOV and it is within a close distance to the sensor. Hence, some embodiments then proceed by continuing to detect the TOFs and use weights appropriate for near objects, such as weights from a column on the left in
At stage 1108 no object has been reliably detected in the FOV. There are at least two reasons for this: there is no object in the FOV, or there is an object in the FOV but it was not detected based on the histogram formed during the initial set of PRIs. The latter case may be due to the weights used, which emphasize TOFs of near objects, and/or the object being at a far distance in the FOV.
The embodiments used in the case at stage 1108 select a new weight set to use with TOFs detected during subsequent PRIs past the DP. The new weight set can be chosen to emphasize TOFs arising from the middle section of the sequence of PRIs. For example, the new weights may be from a column to the right in
The list of weights 1202 is from a column to the left in
The list of weights 1204 is from a column to the right in
In a line scan system, the pixel array 1302 can be read out in a scan-by-scan readout process. The pixel array 1302 can also be read out on a section-by-section basis in systems using a moving illumination source. In other words, the SPADs in the pixels 1304 of various sections of the pixel array 1302 (e.g., two or three columns of pixels) are enabled at respective times. In some embodiments, there may be multiple readouts of individual pixels (e.g., for individual pulse-based measurements) within a scan operation, so that readout is not performed section-by-section. A representative reflected light beam 1318 is shown impinging on the pixel array 1302, and the pixels with the cross-hatching represent the pixels whose SPADs are enabled to detect the photons in the reflected light beam 1318.
The SPAD may be in one of at least three states: an “enabled,” or “activated,” state in which the SPAD section is reverse biased sufficiently so that any received photon can trigger an avalanche output signal; an idle state in which power is supplied to the components of the pixel but not to the extent that a photon impinging the SPAD can trigger an output signal; and an unpowered state (e.g., no reverse bias is applied to the SPAD). A disabled state encompasses both the idle and unpowered states.
The pixels with enabled SPADs constitute a subset of the pixels in the pixel array 1302. The pixels with enabled SPADs are arranged in a non-linear arrangement in
The AF circuitry 1306 is configured to perform row-level signal readout for the pixel array 1302. The AF circuitry 1306 includes N AF circuits for each line of pixels (e.g., a row) in the pixel array 1302, where N represents the number of pixels read out at any given time in the line(s) of pixels. Thus, for the embodiment shown in
The TDC array circuitry 1308 includes N TDC circuits for each line of pixels (e.g., a row) in the pixel array 1302, where N represents the number of pixels that are read out at any given time in the line(s) of pixels. Thus, for the embodiment shown in
The TDC circuits measure the arrival time of the photons impinging on the pixels to be read out (e.g., the pixels that are cross-hatched). Thus, the TDC output values represent the arrival times of the photons. As described earlier, the TDC output values are used to construct histograms for each SPAD in the pixels read out.
The spatial match circuitry 1310 includes N spatial match filter circuits for each line of pixels (e.g., a row) in the pixel array 1302, where N represents the number of pixels that are read out at any given time in the line(s) of pixels. Thus, for the embodiment shown in
In one embodiment, the memory 1312 stores N histograms in N histogram memories 1314 for each line of pixels (e.g., a row) in the pixel array 1302, where N represents the number of pixels read out at any given time in the line(s) of pixels. In other embodiments, the number of histograms for each row may be N+Z, where Z represents one or more additional histogram memories 1314.
The controller 1316 can generate timing signals for the TDC array circuitry 1308. Any suitable controller can be used. For example, the controller 1316 may include ancillary circuitry to generate reference timing signals such as a phase-locked loop circuit or a delay-locked loop circuit.
The SMF memory 1404 is illustrated in more detail in the expanded block. The TDC output value received from line 1406 and a spatial address received from line 1402 are used to access or read out a particular weight in the SMF memory 1404. The spatial address may be used to indicate the current PRI for the reflected light beam striking the pixel array. The line 1402 is distributed to all of the SMF circuits operably connected to the pixel array so the position of a SPAD output within a PRI can be determined. For example, in the embodiment shown in
A weight that is read out of the SMF memory 1404 is received by the add circuit 1410 on line 1408. A histogram is stored in the histogram memory 1412. The add circuit 1410 determines the value (e.g., weighted TDC output value) that is used to increment the count in a bin of the histogram that is associated with the TDC output value on line 1406. In one embodiment, the increment value is the weight multiplied by one. The current count in the bin is read out of the histogram memory 1412 and the current count is incremented by the value. The incremented count is then written back to the histogram memory 1412. An example histogram that is stored in the histogram memory 1412 is visually displayed in
In
As the section 1500 in this example spans 2000 PRIs, the subsections each span 250 respective PRIs from section 1500. The weights applied by the SMF weight multipliers SMFW0, . . . , SMFW7 are taken from the column of weights for the determined TDC histogram bin number. The eight weights can provide a simplified step-wise approximation to an ideal intensity profile or function for the weights, as will be explained further with respect to
Although
In
The one or more processing devices 1704 can control some or all of the operations of the electronic device 1700. The processing device(s) 1704 can communicate, either directly or indirectly, with substantially all of the components of the electronic device 1700. For example, one or more system buses 1712 or other communication mechanisms can provide communication between the SPAD detector(s) 1702, the processing device(s) 1704, the memory 1706, the network interface 1708, and/or the power source 1710. In some embodiments, the processing device(s) 1704 can be configured to receive output signals from the SPAD detectors 1702 and process the output signals to determine one or more characteristics associated with the reflected light, the object (e.g., object 106 in
The processing device(s) 1704 can be implemented as any electronic device capable of processing, receiving, or transmitting data or instructions. For example, the one or more processing devices 1904 can be a microprocessor, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or combinations of multiple such devices. As described herein, the term “processor” is meant to encompass a single processor or processing unit, multiple processors, multiple processing units, or other suitably configured computing element or elements.
The memory 1706 can store electronic data that can be used by the electronic device 1700. For example, the memory 1706 can store electrical data or content such as, for example, audio files, document files, timing and control signals, and so on. The memory 1706 can be configured as any type of memory. By way of example only, memory 1706 can be implemented as random access memory, read-only memory, Flash memory, removable memory, or other types of storage elements, in any combination.
The network interface 1708 can receive data from a user or one or more other electronic devices. Additionally, the network interface 1708 can facilitate transmission of data to a user or to other electronic devices. The network interface 1708 can receive data from a network or send and transmit electronic signals via a wireless or wired connection. For example, the photon counts that are determined by the processing device(s) 1704 can be transmitted to another electronic device using the network interface 1708.
Examples of wireless and wired connections include, but are not limited to, cellular, Wi-Fi, Bluetooth, and Ethernet. In one or more embodiments, the network interface 1708 supports multiple network or communication mechanisms. For example, the network interface 1708 can pair with another device over a Bluetooth network to transfer signals to the other device while simultaneously receiving signals from a Wi-Fi or other wired or wireless connection.
The one or more power sources 1710 can be implemented with any device capable of providing energy to the electronic device 1700. For example, the power source 1710 can be a battery. Additionally or alternatively, the power source 1710 can be a wall outlet that the electronic device 1700 connects to with a power cord. Additionally or alternatively, the power source 1710 can be another electronic device that the electronic device 1700 connects to via a wireless or wired connection (e.g., a connection cable), such as a Universal Serial Bus (USB) cable.
In some embodiments, the SPAD detector 1702 is configured as a back-illuminated SPAD detector. In such embodiments, the pixel array is positioned adjacent to a light receiving surface of the SPAD detector and the circuitry (e.g., gating transistor, quenching transistor, etc.) connected to the SPADs in the pixel array are positioned below the pixel array. Other embodiments can configure the SPAD detector 1702 differently.
The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the described embodiments. Thus, the foregoing descriptions of the specific embodiments described herein are presented for purposes of illustration and description. They are not targeted to be exhaustive or to limit the embodiments to the precise forms disclosed. It will be apparent to one of ordinary skill in the art that many modifications and variations are possible in view of the above teachings.
This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application No. 62/451,031, filed on Jan. 26, 2017, and entitled “Spatial Temporal Weighing In An SPAD Detector,” which is incorporated by reference as if fully disclosed herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4686572 | Takatsu | Aug 1987 | A |
| 4686648 | Fossum | Aug 1987 | A |
| 5105264 | Erhardt et al. | Apr 1992 | A |
| 5329313 | Keith | Jul 1994 | A |
| 5396893 | Oberg et al. | Mar 1995 | A |
| 5471515 | Fossum et al. | Nov 1995 | A |
| 5541402 | Ackland | Jul 1996 | A |
| 5550677 | Schofield et al. | Aug 1996 | A |
| 5781312 | Noda | Jul 1998 | A |
| 5841126 | Fossum et al. | Nov 1998 | A |
| 5880459 | Pryor et al. | Mar 1999 | A |
| 5949483 | Fossum et al. | Sep 1999 | A |
| 6008486 | Stam et al. | Dec 1999 | A |
| 6040568 | Caulfield et al. | Mar 2000 | A |
| 6233013 | Hosier et al. | May 2001 | B1 |
| 6348929 | Acharya et al. | Feb 2002 | B1 |
| 6448550 | Nishimura | Sep 2002 | B1 |
| 6528833 | Lee et al. | Mar 2003 | B2 |
| 6541751 | Bidermann | Apr 2003 | B1 |
| 6670904 | Yakovlev | Dec 2003 | B1 |
| 6713796 | Fox | Mar 2004 | B1 |
| 6714239 | Guidash | Mar 2004 | B2 |
| 6798453 | Kaifu | Sep 2004 | B1 |
| 6816676 | Bianchi et al. | Nov 2004 | B2 |
| 6905470 | Lee et al. | Jun 2005 | B2 |
| 6931269 | Terry | Aug 2005 | B2 |
| 6956605 | Hashimoto | Oct 2005 | B1 |
| 6982759 | Goto | Jan 2006 | B2 |
| 7075049 | Rhodes | Jul 2006 | B2 |
| 7084914 | Blerkom | Aug 2006 | B2 |
| 7091466 | Bock | Aug 2006 | B2 |
| 7119322 | Hong | Oct 2006 | B2 |
| 7133073 | Neter | Nov 2006 | B1 |
| 7259413 | Rhodes | Aug 2007 | B2 |
| 7262401 | Hopper et al. | Aug 2007 | B2 |
| 7271835 | Iizuka | Sep 2007 | B2 |
| 7271836 | Iizuka et al. | Sep 2007 | B2 |
| 7282028 | Kim et al. | Oct 2007 | B2 |
| 7319218 | Krymski | Jan 2008 | B2 |
| 7332786 | Altice et al. | Feb 2008 | B2 |
| 7390687 | Boettiger et al. | Jun 2008 | B2 |
| 7415096 | Sherman | Aug 2008 | B2 |
| 7437013 | Anderson | Oct 2008 | B2 |
| 7443421 | Stavely et al. | Oct 2008 | B2 |
| 7446812 | Ando et al. | Nov 2008 | B2 |
| 7453131 | Marshall et al. | Nov 2008 | B2 |
| 7471315 | Silsby et al. | Dec 2008 | B2 |
| 7502054 | Kalapathy et al. | Mar 2009 | B2 |
| 7525168 | Hsieh | Apr 2009 | B2 |
| 7554067 | Zarnowski et al. | Jun 2009 | B2 |
| 7555158 | Park et al. | Jun 2009 | B2 |
| 7589316 | Dunki-Jacobs | Sep 2009 | B2 |
| 7622699 | Sakakibara | Nov 2009 | B2 |
| 7626626 | Panicacci | Dec 2009 | B2 |
| 7636109 | Nakajima et al. | Dec 2009 | B2 |
| 7667400 | Goushcha | Feb 2010 | B1 |
| 7671435 | Ahn | Mar 2010 | B2 |
| 7696483 | Tkaczyk et al. | Apr 2010 | B2 |
| 7714292 | Agarwal et al. | May 2010 | B2 |
| 7728351 | Shim | Jun 2010 | B2 |
| 7733402 | Egawa | Jun 2010 | B2 |
| 7742090 | Street et al. | Jun 2010 | B2 |
| 7764312 | Ono et al. | Jul 2010 | B2 |
| 7773138 | Lahav et al. | Aug 2010 | B2 |
| 7786543 | Hsieh | Aug 2010 | B2 |
| 7796171 | Gardner | Sep 2010 | B2 |
| 7817198 | Kang et al. | Oct 2010 | B2 |
| 7838956 | McCarten et al. | Nov 2010 | B2 |
| 7873236 | Li et al. | Jan 2011 | B2 |
| 7880785 | Gallagher | Feb 2011 | B2 |
| 7884402 | Ki | Feb 2011 | B2 |
| 7906826 | Martin et al. | Mar 2011 | B2 |
| 7952121 | Arimoto et al. | Mar 2011 | B2 |
| 7952635 | Lauxtermann | May 2011 | B2 |
| 7982789 | Watanabe et al. | Jul 2011 | B2 |
| 8026966 | Altice | Sep 2011 | B2 |
| 8032206 | Farazi et al. | Oct 2011 | B1 |
| 8089036 | Manabe | Jan 2012 | B2 |
| 8089524 | Urisaka | Jan 2012 | B2 |
| 8094232 | Kusaka | Jan 2012 | B2 |
| 8116540 | Dean et al. | Feb 2012 | B2 |
| 8140143 | Picard et al. | Mar 2012 | B2 |
| 8153947 | Barbier et al. | Apr 2012 | B2 |
| 8159570 | Negishi | Apr 2012 | B2 |
| 8159588 | Boemler | Apr 2012 | B2 |
| 8164669 | Compton et al. | Apr 2012 | B2 |
| 8174595 | Honda et al. | May 2012 | B2 |
| 8184188 | Yaghmal | May 2012 | B2 |
| 8194148 | Doida | Jun 2012 | B2 |
| 8194165 | Border et al. | Jun 2012 | B2 |
| 8222586 | Lee | Jul 2012 | B2 |
| 8227844 | Adkisson et al. | Jul 2012 | B2 |
| 8233071 | Takeda | Jul 2012 | B2 |
| 8259228 | Wei et al. | Sep 2012 | B2 |
| 8310577 | Neter | Nov 2012 | B1 |
| 8324553 | Lee | Dec 2012 | B2 |
| 8338856 | Tai et al. | Dec 2012 | B2 |
| 8340407 | Kalman | Dec 2012 | B2 |
| 8350940 | Smith et al. | Jan 2013 | B2 |
| 8355117 | Niclass | Jan 2013 | B2 |
| 8388346 | Rantala et al. | Mar 2013 | B2 |
| 8400546 | Itano et al. | Mar 2013 | B2 |
| 8456540 | Egawa | Jun 2013 | B2 |
| 8456559 | Yamashita et al. | Jun 2013 | B2 |
| 8508637 | Han et al. | Aug 2013 | B2 |
| 8514308 | Itonaga et al. | Aug 2013 | B2 |
| 8520913 | Dean et al. | Aug 2013 | B2 |
| 8546737 | Tian et al. | Oct 2013 | B2 |
| 8547388 | Cheng | Oct 2013 | B2 |
| 8575531 | Hynecek et al. | Nov 2013 | B2 |
| 8581992 | Hamada | Nov 2013 | B2 |
| 8594170 | Mombers et al. | Nov 2013 | B2 |
| 8619163 | Ogura | Dec 2013 | B2 |
| 8619170 | Mabuchi | Dec 2013 | B2 |
| 8629484 | Ohri et al. | Jan 2014 | B2 |
| 8634002 | Kita | Jan 2014 | B2 |
| 8637875 | Finkelstein | Jan 2014 | B2 |
| 8648947 | Sato et al. | Feb 2014 | B2 |
| 8653434 | Johnson et al. | Feb 2014 | B2 |
| 8723975 | Solhusvik | May 2014 | B2 |
| 8724096 | Gosch et al. | May 2014 | B2 |
| 8730345 | Watanabe | May 2014 | B2 |
| 8754983 | Sutton | Jun 2014 | B2 |
| 8755854 | Addison et al. | Jun 2014 | B2 |
| 8759736 | Yoo | Jun 2014 | B2 |
| 8760413 | Peterson et al. | Jun 2014 | B2 |
| 8767104 | Makino et al. | Jul 2014 | B2 |
| 8803990 | Smith | Aug 2014 | B2 |
| 8810703 | Machida | Aug 2014 | B2 |
| 8817154 | Manabe et al. | Aug 2014 | B2 |
| 8874377 | Sickenberger | Oct 2014 | B1 |
| 8879686 | Okada | Nov 2014 | B2 |
| 8902330 | Theuwissen | Dec 2014 | B2 |
| 8902341 | Mabuchi | Dec 2014 | B2 |
| 8908073 | Minagawa et al. | Dec 2014 | B2 |
| 8923994 | Laikari et al. | Dec 2014 | B2 |
| 8934030 | Kim et al. | Jan 2015 | B2 |
| 8936552 | Kateraas et al. | Jan 2015 | B2 |
| 8946610 | Iwabuchi et al. | Feb 2015 | B2 |
| 8982237 | Chen | Mar 2015 | B2 |
| 9006641 | Drader | Apr 2015 | B2 |
| 9017748 | Spelman et al. | Apr 2015 | B2 |
| 9041837 | Li | May 2015 | B2 |
| 9054009 | Oike et al. | Jun 2015 | B2 |
| 9058081 | Baxter | Jun 2015 | B2 |
| 9066017 | Geiss | Jun 2015 | B2 |
| 9066660 | Watson et al. | Jun 2015 | B2 |
| 9088727 | Trumbo | Jul 2015 | B2 |
| 9094623 | Kawaguchi | Jul 2015 | B2 |
| 9099604 | Roy et al. | Aug 2015 | B2 |
| 9100597 | Hu | Aug 2015 | B2 |
| 9106859 | Kizuna et al. | Aug 2015 | B2 |
| 9131171 | Aoki | Sep 2015 | B2 |
| 9151829 | Campbell | Oct 2015 | B2 |
| 9154750 | Pang | Oct 2015 | B2 |
| 9160949 | Zhang et al. | Oct 2015 | B2 |
| 9164144 | Dolinsky | Oct 2015 | B2 |
| 9176241 | Frach | Nov 2015 | B2 |
| 9178100 | Webster et al. | Nov 2015 | B2 |
| 9209320 | Webster | Dec 2015 | B1 |
| 9225948 | Hasegawa | Dec 2015 | B2 |
| 9232150 | Kleekajai et al. | Jan 2016 | B2 |
| 9232161 | Suh | Jan 2016 | B2 |
| 9235267 | Burrough et al. | Jan 2016 | B2 |
| 9270906 | Peng et al. | Feb 2016 | B2 |
| 9276031 | Wan | Mar 2016 | B2 |
| 9277144 | Kleekajai et al. | Mar 2016 | B2 |
| 9287304 | Park et al. | Mar 2016 | B2 |
| 9288380 | Nomura | Mar 2016 | B2 |
| 9288404 | Papiashvili | Mar 2016 | B2 |
| 9293500 | Sharma et al. | Mar 2016 | B2 |
| 9312401 | Webster | Apr 2016 | B2 |
| 9313434 | Dutton et al. | Apr 2016 | B2 |
| 9319611 | Fan | Apr 2016 | B2 |
| 9331116 | Webster | May 2016 | B2 |
| 9344649 | Bock | May 2016 | B2 |
| 9354332 | Zwaans et al. | May 2016 | B2 |
| 9380245 | Guidash | Jun 2016 | B1 |
| 9392237 | Toyoda | Jul 2016 | B2 |
| 9417326 | Niclass et al. | Aug 2016 | B2 |
| 9431439 | Soga et al. | Aug 2016 | B2 |
| 9438258 | Yoo | Sep 2016 | B1 |
| 9445018 | Fettig et al. | Sep 2016 | B2 |
| 9448110 | Wong | Sep 2016 | B2 |
| 9450007 | Motta et al. | Sep 2016 | B1 |
| 9451887 | Watson et al. | Sep 2016 | B2 |
| 9467553 | Heo et al. | Oct 2016 | B2 |
| 9473706 | Malone et al. | Oct 2016 | B2 |
| 9478030 | Lecky | Oct 2016 | B1 |
| 9479688 | Ishii | Oct 2016 | B2 |
| 9490285 | Itonaga | Nov 2016 | B2 |
| 9497397 | Kleekajai et al. | Nov 2016 | B1 |
| 9503616 | Taniguchi et al. | Nov 2016 | B2 |
| 9516244 | Borowski | Dec 2016 | B2 |
| 9538106 | McMahon et al. | Jan 2017 | B2 |
| 9549099 | Fan | Jan 2017 | B2 |
| 9560339 | Borowski | Jan 2017 | B2 |
| 9584743 | Lin et al. | Feb 2017 | B1 |
| 9596420 | Fan et al. | Mar 2017 | B2 |
| 9596423 | Molgaard | Mar 2017 | B1 |
| 9639063 | Dutton et al. | May 2017 | B2 |
| 9661308 | Wang et al. | May 2017 | B1 |
| 9685576 | Webster | Jun 2017 | B2 |
| 9686485 | Agranov et al. | Jun 2017 | B2 |
| 9700240 | Letchner et al. | Jul 2017 | B2 |
| 9741754 | Li et al. | Aug 2017 | B2 |
| 9749556 | Fettig et al. | Aug 2017 | B2 |
| 9774318 | Song | Sep 2017 | B2 |
| 9781368 | Song | Oct 2017 | B2 |
| 9831283 | Shepard et al. | Nov 2017 | B2 |
| 9857469 | Oggier et al. | Jan 2018 | B2 |
| 9870053 | Modarres et al. | Jan 2018 | B2 |
| 9888198 | Mauritzson et al. | Feb 2018 | B2 |
| 9894304 | Smith | Feb 2018 | B1 |
| 9912883 | Agranov et al. | Mar 2018 | B1 |
| 9915733 | Fried | Mar 2018 | B2 |
| 9935231 | Roehrer | Apr 2018 | B2 |
| 9939316 | Scott et al. | Apr 2018 | B2 |
| 9952323 | Deane | Apr 2018 | B2 |
| 9973678 | Mandelli et al. | May 2018 | B2 |
| 9985163 | Moore | May 2018 | B2 |
| 10026772 | Shinohara | Jul 2018 | B2 |
| 10067224 | Moore | Sep 2018 | B2 |
| 10120446 | Pance et al. | Nov 2018 | B2 |
| 10136090 | Vogelsang et al. | Nov 2018 | B2 |
| 10145678 | Wang et al. | Dec 2018 | B2 |
| 10153310 | Zhang et al. | Dec 2018 | B2 |
| 10217889 | Dhulla et al. | Feb 2019 | B2 |
| 10267901 | Drader | Apr 2019 | B2 |
| 10285626 | Kestelli et al. | May 2019 | B1 |
| 10305247 | Bills et al. | May 2019 | B2 |
| 10334181 | Guenter | Jun 2019 | B2 |
| 10338221 | Lee et al. | Jul 2019 | B2 |
| 10416293 | Buckley | Sep 2019 | B2 |
| 10495736 | Zhuang | Dec 2019 | B2 |
| 10651332 | Moussy | May 2020 | B2 |
| 10802148 | Lee et al. | Oct 2020 | B2 |
| 20030036685 | Goodman et al. | Feb 2003 | A1 |
| 20040207836 | Chhibber et al. | Oct 2004 | A1 |
| 20050026332 | Fratti et al. | Feb 2005 | A1 |
| 20060274161 | Ing et al. | Dec 2006 | A1 |
| 20070263099 | Motta et al. | Nov 2007 | A1 |
| 20080177162 | Bae et al. | Jul 2008 | A1 |
| 20080315198 | Jung | Dec 2008 | A1 |
| 20090096901 | Bae et al. | Apr 2009 | A1 |
| 20090101914 | Hirotsu et al. | Apr 2009 | A1 |
| 20090146234 | Luo et al. | Jun 2009 | A1 |
| 20090201400 | Zhang et al. | Aug 2009 | A1 |
| 20090219266 | Lim et al. | Sep 2009 | A1 |
| 20100110018 | Faubert et al. | May 2010 | A1 |
| 20100134631 | Voth | Jun 2010 | A1 |
| 20100159632 | Rhodes et al. | Jun 2010 | A1 |
| 20110080500 | Wang et al. | Apr 2011 | A1 |
| 20110156197 | Tivarus et al. | Jun 2011 | A1 |
| 20110164162 | Kato | Jul 2011 | A1 |
| 20120092541 | Tuulos et al. | Apr 2012 | A1 |
| 20120162632 | Dutton | Jun 2012 | A1 |
| 20130147981 | Wu | Jun 2013 | A1 |
| 20140049683 | Guenter et al. | Feb 2014 | A1 |
| 20140071321 | Seyama | Mar 2014 | A1 |
| 20140132528 | Catton | May 2014 | A1 |
| 20140231630 | Rae et al. | Aug 2014 | A1 |
| 20150062391 | Murata | Mar 2015 | A1 |
| 20150277559 | Vescovi et al. | Oct 2015 | A1 |
| 20150312479 | McMahon et al. | Oct 2015 | A1 |
| 20160050379 | Jiang et al. | Feb 2016 | A1 |
| 20160218236 | Dhulla et al. | Jul 2016 | A1 |
| 20160219232 | Murata | Jul 2016 | A1 |
| 20160274237 | Stutz | Sep 2016 | A1 |
| 20160365380 | Wan | Dec 2016 | A1 |
| 20170047363 | Choi et al. | Feb 2017 | A1 |
| 20170052065 | Sharma et al. | Feb 2017 | A1 |
| 20170082746 | Kubota et al. | Mar 2017 | A1 |
| 20170084133 | Cardinali et al. | Mar 2017 | A1 |
| 20170142325 | Shimokawa et al. | May 2017 | A1 |
| 20170223292 | Ikeda et al. | Aug 2017 | A1 |
| 20170272675 | Kobayashi | Sep 2017 | A1 |
| 20170364736 | Ollila | Dec 2017 | A1 |
| 20170373106 | Li et al. | Dec 2017 | A1 |
| 20180090526 | Mandai et al. | Mar 2018 | A1 |
| 20180090536 | Mandai et al. | Mar 2018 | A1 |
| 20180109742 | Agranov et al. | Apr 2018 | A1 |
| 20180209846 | Mandai et al. | Jul 2018 | A1 |
| 20180213205 | Oh | Jul 2018 | A1 |
| 20190018119 | Laifenfeld | Jan 2019 | A1 |
| 20190027674 | Zhang et al. | Jan 2019 | A1 |
| 20190198701 | Moussy | Jun 2019 | A1 |
| 20200278429 | Mandai et al. | Sep 2020 | A1 |
| 20200286946 | Mandai et al. | Sep 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 1630350 | Jun 2005 | CN |
| 1774032 | May 2006 | CN |
| 1833429 | Sep 2006 | CN |
| 1842138 | Oct 2006 | CN |
| 1947414 | Apr 2007 | CN |
| 101189885 | May 2008 | CN |
| 101221965 | Jul 2008 | CN |
| 101233763 | Jul 2008 | CN |
| 101379615 | Mar 2009 | CN |
| 101472059 | Jul 2009 | CN |
| 101567977 | Oct 2009 | CN |
| 101622859 | Jan 2010 | CN |
| 101739955 | Jun 2010 | CN |
| 101754029 | Jun 2010 | CN |
| 101803925 | Aug 2010 | CN |
| 102036020 | Apr 2011 | CN |
| 102067584 | May 2011 | CN |
| 102208423 | Oct 2011 | CN |
| 102451160 | May 2012 | CN |
| 102668542 | Sep 2012 | CN |
| 102820309 | Dec 2012 | CN |
| 102821255 | Dec 2012 | CN |
| 103024297 | Apr 2013 | CN |
| 103051843 | Apr 2013 | CN |
| 103299437 | Sep 2013 | CN |
| 103329513 | Sep 2013 | CN |
| 103546702 | Jan 2014 | CN |
| 103779437 | May 2014 | CN |
| 104103655 | Oct 2014 | CN |
| 104779259 | Jul 2015 | CN |
| 104810377 | Jul 2015 | CN |
| 204761615 | Nov 2015 | CN |
| 105185796 | Dec 2015 | CN |
| 105489624 | Apr 2016 | CN |
| 102010060527 | Apr 2012 | DE |
| 1763228 | Mar 2007 | EP |
| 2023611 | Feb 2009 | EP |
| 2107610 | Oct 2009 | EP |
| 2230690 | Sep 2010 | EP |
| 2512126 | Oct 2012 | EP |
| 2787531 | Oct 2014 | EP |
| S61123287 | Jun 1986 | JP |
| 2007504670 | Aug 1987 | JP |
| 2000059697 | Feb 2000 | JP |
| 2001211455 | Aug 2001 | JP |
| 2001358994 | Dec 2001 | JP |
| 2004111590 | Apr 2004 | JP |
| 2004319576 | Nov 2004 | JP |
| 2005318504 | Nov 2005 | JP |
| 2006287361 | Oct 2006 | JP |
| 2007516654 | Jun 2007 | JP |
| 2008507908 | Mar 2008 | JP |
| 2008271280 | Nov 2008 | JP |
| 2008543061 | Nov 2008 | JP |
| 2009021809 | Jan 2009 | JP |
| 2009159186 | Jul 2009 | JP |
| 2009212909 | Sep 2009 | JP |
| 2009296465 | Dec 2009 | JP |
| 2010080604 | Apr 2010 | JP |
| 2010114834 | May 2010 | JP |
| 2011040926 | Feb 2011 | JP |
| 2011049697 | Mar 2011 | JP |
| 2011091775 | May 2011 | JP |
| 11-216970 | Oct 2011 | JP |
| 11-217315 | Oct 2011 | JP |
| 2011097646 | Dec 2011 | JP |
| 2012010306 | Jan 2012 | JP |
| 2012019516 | Jan 2012 | JP |
| 2012038981 | Feb 2012 | JP |
| 2012513160 | Jun 2012 | JP |
| 2012169530 | Sep 2012 | JP |
| 2013051523 | Mar 2013 | JP |
| 2013070240 | Apr 2013 | JP |
| 2013529035 | Jul 2013 | JP |
| 2014081254 | May 2014 | JP |
| 2014225647 | Dec 2014 | JP |
| 2015041746 | Mar 2015 | JP |
| 2016145776 | Aug 2016 | JP |
| 20030034424 | May 2003 | KR |
| 20030061157 | Jul 2003 | KR |
| 20050103732 | Nov 2005 | KR |
| 20080069851 | Jul 2008 | KR |
| 20100008239 | Jan 2010 | KR |
| 20100065084 | Jun 2010 | KR |
| 20130074459 | Jul 2013 | KR |
| 200520551 | Jun 2005 | TW |
| 200803481 | Jan 2008 | TW |
| 201110689 | Mar 2011 | TW |
| 201301881 | Jan 2013 | TW |
| WO 05041304 | May 2005 | WO |
| WO 06014641 | Feb 2006 | WO |
| WO 06130443 | Dec 2006 | WO |
| WO 07049900 | May 2007 | WO |
| WO 10120945 | Oct 2010 | WO |
| WO 12011095 | Jan 2012 | WO |
| WO 12032353 | Mar 2012 | WO |
| WO 12053363 | Apr 2012 | WO |
| WO 12088338 | Jun 2012 | WO |
| WO 12122572 | Sep 2012 | WO |
| WO 12138687 | Oct 2012 | WO |
| WO 13008425 | Jan 2013 | WO |
| WO 13179018 | Dec 2013 | WO |
| WO 13179020 | Dec 2013 | WO |
| WO 17112416 | Jun 2017 | WO |
| WO 20045123 | Mar 2020 | WO |
| Entry |
|---|
| U.S. Appl. No. 15/699,806, filed Sep. 8, 2017, Li et al. |
| U.S. Appl. No. 15/879,350, filed Jan. 24, 2018, Mandai et al. |
| U.S. Appl. No. 16/226,491, filed Dec. 19, 2018, McMahon. |
| Aoki, et al., “Rolling-Shutter Distortion-Free 3D Stacked Image Sensor with -160dB Parasitic Light Sensitivity In-Pixel Storage Node,” ISSCC 2013, Session 27, Image Sensors, 27.3 27.3 A, Feb. 20, 2013, retrieved on Apr. 11, 2014 from URL:http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6487824. |
| Charbon, et al., SPAD-Based Sensors, TOF Range-Imaging Cameras, F. Remondino and D. Stoppa (eds.), 2013, Springer-Verlag Berlin Heidelberg, pp. 11-38. |
| Cox, “Getting histograms with varying bin widths,” http://www.stata.com/support/faqs/graphics/histograms-with-varying-bin-widths/, Nov. 13, 2017, 5 pages. |
| Elgendi, “On the Analysis of Fingertip Photoplethysmogram Signals,” Current Cardiology Reviews, 2012, vol. 8, pp. 14-25. |
| Feng, et al., “On the Stoney Formula for a Thin Film/Substrate System with Nonuniform Substrate Thickness,” Journal of Applied Mechanics, Transactions of the ASME, vol. 74, Nov. 2007, pp. 1276-1281. |
| Fu, et al., “Heart Rate Extraction from Photoplethysmogram Waveform Using Wavelet Multui-resolution Analysis,” Journal of Medical and Biological Engineering, 2008, vol. 28, No. 4, pp. 229-232. |
| Gallivanoni, et al., “Progress n Quenching Circuits for Single Photon Avalanche Diodes,” IEEE Transactions on Nuclear Science, vol. 57, No. 6, Dec. 2010, pp. 3815-3826. |
| Han, et al., “Artifacts in wearable photoplethysmographs during daily life motions and their reduction with least mean square based active noise cancellation method,” Computers in Biology and Medicine, 2012, vol. 42, pp. 387-393. |
| Jahromi et al., “A Single Chip Laser Radar Receiver with a 9×9 SPAD Detector Array and a 10-channel TDC,” 2013 Proceedings of the ESSCIRC, IEEE, Sep. 14, 2015, pp. 364-367. |
| Leslar, et al., “Comprehensive Utilization of Temporal and Spatial Domain Outlier Detection Methods for Mobile Terrestrial LiDAR Data,” Remote Sensing, 2011, vol. 3, pp. 1724-1742. |
| Lopez-Silva, et al., “Heuristic Algorithm for Photoplethysmographic Heart Rate Tracking During Maximal Exercise Test,” Journal of Medical and Biological Engineering, 2011, vol. 12, No. 3, pp. 181-188. |
| Mota, et al., “A flexible multi-channel high-resolution Time-to-Digital Converter ASIC,” Nuclear Science Symposium Conference Record IEEE, 2000, Engineering School of Geneva, Microelectronics Lab, Geneva, Switzerland, 8 pages. |
| Niclass, et al., “Design and Characterization of a CMOS 3-D Image Sensor Based on Single Photon Avalanche Diodes,” IEEE Journal of Solid-State Circuits, vol. 40, No. 9, Sep. 2005, pp. 1847-1854. |
| Santos, et al., “Accelerometer-assisted PPG Measurement During Physical Exercise Using the LAVIMO Sensor System,” Acta Polytechnica, 2012, vol. 52, No. 5, pp. 80-85. |
| Sarkar, et al., “Fingertip Pulse Wave (PPG signal) Analysis and Heart Rate Detection,” International Journal of Emerging Technology and Advanced Engineering, 2012, vol. 2, No. 9, pp. 404-407. |
| Schwarzer, et al., On the determination of film stress from substrate bending: STONEY'S formula and its limits, Jan. 2006, 19 pages. |
| Shen et al., “Stresses, Curvatures, and Shape Changes Arising from Patterned Lines on Silicon Wafers,” Journal of Applied Physics, vol. 80, No. 3, Aug. 1996, pp. 1388-1398. |
| Shin, et al., “Photon-Efficient Computational 3D and Reflectivity Imaging with Single-Photon Detectors,” IEEE International Conference on Image Processing, Paris, France, Oct. 2014, 11 pages. |
| Tisa, et al., “Variable-Load Quenching Circuit for single-photon avalanche diodes,” Optics Express, vol. 16, No. 3, Feb. 4, 2008, pp. 2232-2244. |
| Ullrich, et al., “Linear LIDAR versus Geiger-mode LIDAR: Impact on data properties and data quality,” Laser Radar Technology and Applications XXI, edited by Monte D. Turner, Gary W. Kamerman, Proc. of SPIE, vol. 9832, 983204, 2016, 17 pages. |
| Yan, et al., “Reduction of motion artifact in pulse oximetry by smoothed pseudo Wigner-Ville distribution,” Journal of NeuroEngineering and Rehabilitation, 2005, vol. 2, No. 3, pp. 1-9. |
| Yousefi, et al., “Adaptive Cancellation of Motion Artifact in Wearable Biosensors,” 34th Annual International Conference of the IEEE EMBS, San Diego, California, Aug./Sep. 2012, pp. 2004-2008. |
| U.S. Appl. No. 16/688,932, filed Nov. 19, 2019, Niclass et al. |
| U.S. Appl. No. 16/876,511, filed May 18, 2020, Mandai et al. |
| U.S. Appl. No. 16/877,121, filed May 18, 2020, Mandai et al. |
| Number | Date | Country | |
|---|---|---|---|
| 62451031 | Jan 2017 | US |
