Patent Application
20240319369
The present disclosure relates to a time-of-flight (ToF) system, and in particular a system which includes a controller, a light source, a first image sensor and a second image sensor. The present disclosure also relates to a method of detecting light incident on the second image sensor.
Time-of-flight (ToF) camera systems are range imaging systems that resolve the distance between the camera and an object by measuring the round trip of light emitted from the ToF camera system and reflected by the object. The systems typically comprise a light source (such as a laser or LED), a light source driver to control the emission of light from the light source, an image sensor to image light reflected by the object, an image sensor driver to control the operation of the image sensor, optics to shape the light emitted from the light source and to focus light reflected by the object onto the image sensor, and a computation unit configured to determine the distance to the object based on the emitted light and the corresponding light reflection from the object.
In a Continuous Wave (CW) ToF camera system (also known as an indirect ToF system), multiple periods of a modulated light are emitted from the laser. The system is then configured to determine the distance to the imaged object based on a phase difference between the emitted light and the received reflected light. CW ToF systems often modulate the emitted laser light at a first modulation frequency and determine a first phase difference between the emitted light and reflected light, before modulating the emitted laser light at a second frequency and determining a further phase difference between the emitted light and reflected light. A depth map/depth frame can then be determined based on the first and second phase differences. The first frequency and second frequency are different so that the first and second phase differences can be used to resolve phase wrapping.
Alternatively, in some continuous wave systems, only a single frequency is used.
The present inventors have recognised that there is need for improved correction of cyclic and offset errors.
In a first aspect, there is provided a time-of-flight, ToF, imaging system, the ToF imaging system comprising: a light source; an image sensor; and a controller coupled to the light source and the image sensor, wherein the controller is configured to: control the light source to emit modulated light using a modulated control signal; determine, based on charge accumulated by the image sensor, an error of the ToF system, wherein the error comprises at least one of a cyclic error and an offset error; and modify the light emitted from the light source to reduce the error of the ToF system.
The controller may be configured to modify the light emitted from the light source by adjusting a timing of the modulated control signal relative to a charge accumulation control signal used to control the image sensor.
The controller may be configured to modify the light emitted from the light source by pre-distorting the modulated control signal.
Pre-distorting the modulated control signal may comprise one or more of: adjusting a duty cycle of the modulated control signal; controlling the light source to emit light using a first modulated control signal for a first period of time and controlling the light source to emit light using a second modulated control signal for a second period of time; adjusting a shape of the modulated control signal.
The ToF system may comprise a light source driver configured to receive the modulated control signal, generate a drive signal based on the modulated control signal and drive the light source to emit light using the drive signal, wherein the controller is configured to modify the light emitted from the light source by adjusting at least one setting of the light source driver so as to adjust the drive signal generated by the light source driver.
At least one setting of the light source driver may comprise one or more of: sag compensation; over/under shoot; up-sampled arbitrary waveform generation; and analog component adjustment.
The ToF imaging system may further comprise a controllable delay circuit in a signal path of the modulated control signal, wherein the controllable delay block is configured to receive the modulated control signal and delay transmission of the modulated control signal to the light source, and wherein the controller is configured to modify the light emitted from the light source by setting the length of the delay applied by the controllable delay block.
The controller may be configured to iteratively modify the light emitted from the light source until the error of the ToF system is reduced.
The ToF system may further comprise an internal optical path for carrying part of the modulated light emitted from the light source to at least part of the imaging sensor, wherein the controller is configured to determine the error of the ToF system based on the charge accumulated by the imaging sensor as a result of light carried from the light source to the imaging sensor by the internal optical path.
The imaging sensor may comprise a primary image sensor area for receiving light for use in determining a ToF image; and a secondary image sensor for receiving light from the internal optical path, wherein the controller is configured to determine the error of the ToF system based on the charge accumulated by the secondary image sensor.
In a second aspect there is provided a method of operating a time-of-flight (ToF) system, the method comprising: emitting light from at least one light source; detecting light emitted from the at least one light source on a secondary image sensor; determining one or more of an offset and cyclic error present in the ToF system based on the light detected at the secondary image sensor; compensating for one or more of the offset and cyclic error determined by modifying a signal input to the at least one light source.
The method may further comprise supplying a first modulated control signal and a second modulated control signal to the light source.
The method may further comprise modifying one or more of: a timing, phase offset and duty cycle of the first modulated control signal and the second modulated control signal to compensate for one or more of the cyclic and offset error of the ToF system.
The method may further comprise iteratively modifying one or more of the first modulated control signal and the second modulated control signal until one or more of the cyclic and offset error determined by the processor is reduced.
The method may further comprise modifying a controllable delay in the signal path of the first light source to compensate for the determined offset error.
The method may further comprise determining a distance to an object based on light incident on a primary image sensor.
In a third aspect, there is provided a time-of-flight (ToF) system, the ToF system comprising: a light emission unit comprising: at least one light source configured to emit light; at least one secondary image sensor; an optical path of a known distance between the at least one light source and the at least one secondary image sensor; a processor configured to determine offset and/or cyclic errors of the ToF system based on an output of the secondary image sensor, the processor further configured to alter a control signal input to the light emission unit to compensate for the offset and/or cyclic errors of the ToF system.
The processor may be configured to iteratively adjust the control signal input to the light emission unit until one or more of the determined offset and cyclic error is reduced.
The ToF system may further comprise a primary image sensor.
The ToF system may use multiple secondary image sensors or secondary feedback arrays.
According to a fourth aspect, there is provided a time-of-flight (ToF) imaging system, the ToF system comprising: a light source for outputting modulated light; a main image sensor; a plurality of secondary image sensors; an internal optical path of a known distance for carrying part of the modulated light emitted from the light source to each of the plurality of secondary image sensors; and a controller configured to determine, based on charge accumulated by the plurality of secondary image sensors, an error of the ToF system, wherein the error comprises at least one of a cyclic error and an offset error.
The controller may be configured to control the light source to emit the modulated light for a first amount of time using a first laser modulation signal.
The controller may be configured to control the charge accumulation of the plurality of secondary image sensors during the first amount of time such that each image sensor of the plurality of secondary image sensors is configured to accumulate charge over a different phase range of the first laser modulation signal.
The controller may be configured to control the light source to emit light for a second amount of time using a second laser modulation signal.
The second laser modulation signal may be a delayed version of the first laser modulation signal.
The second laser modulation signal may be the same as the first laser modulation signal.
The controller may be configured to control the charge accumulation of the plurality of secondary image sensors during the second amount of time such that each image sensor of the plurality of secondary image sensors is configured to accumulate charge over a different phase range of the second laser modulation signal.
The controller may be configured to determine a phase relationship between modulated light emitted by the light source and light received by the plurality of secondary image sensors.
The controller may be configured to determine the error of the ToF system by comparing the determined phase relationship with a theoretical phase relationship based on the known distance of the internal optical path.
The plurality of secondary image sensors may comprise four image sensors.
The controller may be configured to perform a ToF distance measurement by detecting light incident on the first imaging sensor.
The controller may be configured to correct the ToF measurement using the determined error of the ToF system to reduce the error of the ToF system.
The controller may be configured to use the determined error of the ToF system to modify the light outputted from the light source to reduce the error of the ToF system.
According to a fifth aspect, there is provided a method of determining distance using a time of flight (ToF) system, the method comprising: emitting modulated light using a light source; detecting the emitted light at a main image sensor; detecting the emitted light at a plurality of secondary image sensors located a known distance from the at least one light source; determining at least one of an offset and a cyclic error of the ToF system based on charge accumulated by the plurality of secondary image sensors.
The method may further comprise controlling charge accumulation of the plurality of secondary image sensors during a first time period such that each image sensor of the plurality of secondary image sensors is configured to accumulate charge over a phase range of the emitted light different to that of the other image sensors of the plurality of secondary image sensors.
The method may further comprise controlling charge accumulation of the plurality of image sensors during a second time period such that each image sensor of the plurality of images sensors is configured to accumulate charge over a phase range of the emitted light different to that of the other image sensors of the plurality of secondary image sensors and different to the phase range during the first time period.
Determining at least one of an offset and cyclic error may comprise determining a phase relationship between the emitted modulated light and light received by the plurality of secondary image sensors; and comparing a determined phase relationship with a theoretical phase relationship based on the known distance.
The method may further comprise applying a calibration to the ToF system based on the determined at least one of offset and cyclic error.
The method may further comprise determining a ToF distance measurement based on the light detected at the main image sensor.
According to a sixth aspect, there is provided a time-of-flight (ToF) imaging system, the ToF system comprising: a light source configured to emit light; a plurality of secondary image sensors; an optical path of a known distance between the at least one light source and the plurality of secondary image sensors, arranged such that part of the emitted light is carried by the optical path to the plurality of secondary image sensors; and a controller configured to: control the plurality of secondary image sensors, such that each image sensor of the plurality of secondary image sensors accumulates charge over a different phase range of the emitted light; and determine at least one of a phase error and an offset error, using charge accumulated by each of the plurality of secondary image sensors.
The ToF system may utilise a secondary array to perform further functions.
According to a seventh aspect, there is provided a time-of-flight, ToF, imaging system, the ToF system comprising: a first light source for emitting first light; a first image sensor; a second image sensor; an internal optical path for carrying part of the first light from the first light source to the second image sensor; and a controller configured to perform a ToF measurement by detecting light incident on the first imaging sensor The controller may be configured to determine a characteristic of the light received at the second image sensor.
The characteristic may comprise one or more of an instantaneous intensity of the light, a measure of the intensity of light received over a period of time, an amplitude of the light, a linearity of the light.
The controller may be configured to compare the characteristic of the light received at the second image sensor with a threshold range; operate a safety function if the characteristic of light is outside the threshold range.
The system may further comprise an optical control switch located in the internal optical path between the first light source and the second image sensor, wherein the optical control switch is controllable to allow or prevent transmission of light along the internal optical path.
The controller may be configured to control the optical control switch to allow transmission along the optical path during a period in which the controller is performing an operation which uses light detected at the second image sensor.
The controller may be configured to control the optical control switch to prevent transmission along the optical path during a period in which the controller is performing an operation which does not use light detected at the second image sensor.
The ToF system may further comprise a second light source.
The controller may be configured to determine at least one of an offset error and a cyclic error of the first light source; and determine at least one of an offset error and a cyclic error of the second light source.
The ToF system may further comprise a controllable delay in a signal path of the first light source or the second light source, and wherein the controllable delay is configured to compensate a difference between the offset error of the first light source and the second light source.
The second image sensor may be part of the first image sensor.
The internal optical path may comprise an optical guide configured to direct part of the emitted fist light from the first light source to the second image sensor. The optical guide may comprise one or more of: an optical fiber; a mirror; a lens; a refractive element.
The controller may be configured to determine at least one of a cyclic error and offset error, based on charge accumulated by the second image sensor as a result of the light carried from the first light source to the second image sensor by the internal optical path.
According to an eight aspect, there is provided a method of controlling a time-of-flight imaging system, the method comprising: emitting first light from a first light source; performing a ToF measurement based on detecting light incident on a first image sensor; detecting light incident on a second image sensor, the light detected on the second image sensor having travelled down an optical path of a known distance from the first light source.
The method may further comprise determining a characteristic of the light incident on the second image sensor; and comparing the characteristic of the light incident on the second image sensor to a threshold range to determine whether the first light source is operating within a predetermined operating range.
The method may further comprise controlling an optical switch in the optical path between the first light source and the second image sensor, the optical switch capable of preventing light transmission between the first light source and the second image sensor.
According to a ninth aspect, there is provided, a time-of-flight, ToF, imaging system, the ToF system comprising: a light emission unit comprising at least one light source; a first image sensor; an optical guide configured to direct light along an optical path of a known distance between the at least one light source and the first image sensor; an optical switch, located in the optical path between the at least one light source and the first image sensor, wherein the optical switch is configurable to allow or prevent light transmission down the optical path between the at least one light source and the first image sensor; and a controller configured to: control the optical switch to allow light transmission along the optical path; and determine at least one of a cyclic error and an offset error of the ToF imaging system based on detecting light incident on the first image sensor.
When the controller is not determining the cyclic error or offset error, the controller may be configured to control the optical switch to prevent light transmission along the optical path.
The controller may be configured to perform a time-of-flight measurement using light incident on the first imaging sensor, wherein when the controller is performing a time-of-flight measurement, the optical switch is controlled to prevent light transmission along the optical path.
A CW ToF camera system uses an image sensor to determine the phase shift between emitted and detected light. This phase shift may be used to determine the distance travelled by the emitted light. However, cyclic and offset errors may be present in CW ToF systems, lowering the accuracy of the determined distance measurement. To address this issue, the inventors have developed various different ways in which a secondary image sensor may be used to determine the cyclic and/or offset errors. The use of a second imaging sensor for error determination may allow a number of further features to be implemented. For example, the second image sensor may allow for monitoring of the safety of the optical system and/or, where multiple light sources are used, allow the detection and correction of a skew between the light sources. Further, the inventors have determined a number of ways in which the determined offset and cyclic errors may be compensated.
The system 100 also comprises an imaging sensor 120 that comprises a plurality (in this case m×n) of imaging pixels. The imaging pixels can be at least one of various demodulating types. A converter system 130 (comprising a plurality of amplifiers and ADCs) is coupled to the imaging sensor 120 for reading off charge accumulated on the imaging pixels and converting to digital values, which are output to the memory processor & controller 140. The nature of the values read out from the imaging sensor 120 will depend on the technology of the imaging sensor 120. For example, if the imaging sensor is a CMOS sensor, voltage values may be readout, where each voltage value is dependent on the charge accumulated in an imaging pixel of the imaging sensor 120, such that the readout values are each indicative of charge accumulated in imaging pixels of the imaging sensor 120. In other sensor technologies, the nature of the readout values may be different, for example charge may be directly readout, or current, etc. The memory processor & controller 140 is configured to determine depth frames (also referred to as depth maps), indicative of distance to the object being imaged, based on the received digital values indicative of charge accumulated on the imaging pixels. The memory processor & controller 140 may also be configured to determine active brightness frames (also referred to as 2D IR frames/images). The memory processor & controller 140 controls a clock generation circuit 150, which outputs timing signals for driving the laser 110 and for controlling the imaging sensor 120. The converter system 130, memory processor & controller 140 and clock generation circuit 150 may together be referred to as an image acquisition system, configured to determine one or more depth frames by controlling the laser 110 and image sensor 120, reading off the image sensor 120 and processing the resultant data.
During a subsequent read out period of time 2201, the memory processor & controller 140 and clock generation circuit 150 control the first laser 1101 to cease emitting light and control the imaging sensor 120 to readout image sensor values that are indicative of the charge accumulated in the imaging pixels of the imaging sensor 120. The nature of the readout values will depend on the technology of the imaging sensor 120. For example, if the imaging sensor 120 is a CMOS sensor, voltage values may be readout, where each voltage value is dependent on the charge accumulated in an imaging pixel of the imaging sensor 120, such that the readout values are each indicative of charge accumulated in imaging pixels of the imaging sensor 120. In other sensor technologies, the nature of the readout values may be different, for example charge may be directly readout, or current, etc. For example, the imaging sensor 120 may be controlled to readout image sensor values row-by-row using any standard readout process and circuitry well understood by the skilled person. In this way, a sample of charge accumulated by each imaging pixel during the period 2101 may be read off the imaging sensor 120, converted to a digital value and then stored by the memory processor & controller 140. The group of values, or data points, arrived at the conclusion of this process is referred to in this disclosure as a charge sample.
It will be appreciated that the accumulation period of time 2101 may last for multiple periods/cycles of the first modulation signal (as can be seen in
During accumulation period of time 2102, the memory processor & controller 140 and clock generation circuit 150 again control the first laser 1101 to output first laser light modulated by the first modulation signal for an accumulation period of time 2102. This is very similar to the accumulation period 2101, except during accumulation period of time 2102 the memory processor & controller 140 and clock generation circuit 150 controls the imaging sensor 120 to accumulate charge for the second part/interval of the period/cycle of the first modulation signal (90° to 270°, or π/2 to 3π/2). The read-out period 2202 is very similar to period 2201, except the obtained charge sample relates to a shifted or delayed interval of π/2 to 3π/2 of the first modulation signal.
Accumulation period of time 2103 is very similar to the period 2102, except the memory processor & controller 140 and clock generation circuit 150 controls the imaging sensor 120 to accumulate charge for the third part/interval of the period/cycle of the first modulation signal (180° to 360°, or π to 2π). The read-out period 2203 is very similar to period 2202, except the sampled charge data relates to a shifted or delayed interval of π to 2π of the first modulation signal.
Finally, accumulation period of time 2104 is very similar to the period 2103, except the memory processor & controller 140 and clock generation circuit 150 also controls the imaging sensor 120 to accumulate charge based on the incident reflected first laser light for a fourth part/interval of the period/cycle of the first modulation signal (270° to 90°, or 3π/2 to π/2). The read-out period 2204 is very similar to period 2203, except the charge sample relates to a shifted or delayed interval of 3π/2 to π/2 (or, put another, a shifted or delayed interval of 3π/2 to 5π/2).
It can be seen from the above that for each accumulation period 2101-2104, the start timing of pixel accumulation timing relative to the laser modulation signal is shifted (i.e., the relative phase of the laser modulation signal and the pixel demodulation signal, which controls pixel accumulation timing, is shifted). This may be achieved either by adjusting the pixel demodulation signal or by adjusting the laser modulation signal. For example, the timing of the two signals may be set by a clock and for each of the accumulation periods 2101-2104, either the laser modulation signal or the pixel demodulation signal may be incrementally delayed by π/2.
Whilst in this example each accumulation period 2101-2104 lasts for 50% of the period of the laser modulation signal (i.e., for 180°), in an alternative each accumulation period may be shorter, for example 60°, or 90°, or 120°, etc, with the start of each accumulation period relatively offset by 90° as explained above.
After completing this, four samples of data (charge samples) have been acquired and stored in memory. They together may be referred to as a first set of charge samples. Immediately after the read out period 2204, or at some later time, a phase relationship between the first laser light and the received reflected light may be determined using the four charge samples (for example by performing a discrete Fourier transform (DFT) on the samples to find the real and imaginary parts of the fundamental frequency, and then determining the phase from the real and imaginary parts, as will be well understood by the skilled person). This may be performed by the image acquisition system, or the charge samples may be output from the image acquisition system to an external processor via a data bus for the determination of the phase relationship. Optionally, active brightness (2D IR) may also be determined (either by the image acquisition system or the external processor) for the reflected first laser light using the four samples (for example, by determining the magnitude of the fundamental frequency from the real and imaginary parts, as will be well understood by the skilled person).
The skilled person will readily understand that using DFT to determine the phase relationship between the first laser light and the received reflected laser light, and to determine active brightness, is merely one example and that any other suitable alternative technique may be used. By way of brief explanation a further non-limiting example is now described.
The transmitted, modulated laser signal may be described by the following equation:
The signal received at the imaging sensor may be described by the following equation:
Accumulation timing of the imaging pixels may be controlled using a demodulation signal, g(t-π), which is effectively a time delayed version of the illumination signal.
The imaging pixels of the imaging sensor 120 effectively multiply the signals r(t) and g(t-τ). The resulting signal may be integrated by the imaging pixels of the imaging sensor to yield a cross correlation signal c(τ):
By driving the imaging sensor to accumulate at different phase offsets during different accumulation periods, as described above, it is possible to measure correlation at different time offsets τ (phase-offsets φ) 0, π/2, π, 3π/2:
From these readings, it can be determined that the phase offset/time of flight can be found by:
Therefore, a depth image or map can be determined using the four charge samples acquired from the image sensor.
An active brightness, or 2D IR, image/frame may also be determined by determining √{square root over ((A4-A2)2+(A1-A3)2)}.
Subsequently, the process described earlier in relation to periods 2101-2104 and 2201-2204 may then be repeated in accumulation periods 2301-2304 and read out periods 2401-2404. These are the same as the accumulation periods 2101-2104 and read out periods 2201-2204, except rather than driving the laser 110 to emit light modulated with the first modulation signal, the laser 110 is driven to emit light modulated with a second modulation signal. The second modulation signal has a second frequency f2, which is higher than the first frequency f1. As a result, four further samples of data (charge samples) are obtained and stored in memory. Based on these charge samples, a phase relationship between the second laser light and the received reflected light (and optionally also the active brightness for the reflected second laser light) may be determined either by the image acquisition system or the external processor, for example using DFT or correlation function processes as described above.
Using the determined phase relationship between the first laser light and the received reflected light and the determined phase relationship between the second laser light and the received reflected light, phase unwrapping may be performed and a single depth image/frame determined by the memory processor & controller 140 (as will be understood by the skilled person). In this way, any phase wrapping issues can be resolved so that an accurate depth frame can be determined. This process may be repeated many times in order to generate a time series of depth frames, which may together form a video.
Optionally, a 2D IR frame may also be determined using the determined active brightness for the first laser light and the determined active brightness for the second laser light.
In this example, there are four accumulation periods for each laser modulation frequency, each accumulation period being at a different phase offset relative to the laser modulation signal. This may be referred to as four times oversampling. However, in a different example there may be a different number of accumulation periods per frequency, such as two, three, six, eight, etc. Typically, larger numbers of different accumulation periods (each at a different phase offset relative to the laser drive signal) reduces the number of harmonics that fold back into the fundamental frequency (as will be understood from Nyquist sampling theory), which in turn reduces noise. This is briefly explained below.
Whilst
To determine the distance to an object, CW ToF systems rely on the phase relationship between the emitted/transmitted laser light and the received reflected light. However, delays are present within the timing of the ToF system. These delays may include process, voltage and temperature (PVT) delays which occur in different parts of the system, for example, in the memory processor & control 140, in the clock generation circuit 150 and in the laser driver 105, among other parts of the system. These errors may vary due to variance in the temperature of the ToF system and its environment, variance in the supply voltage and the process variance between different ToF systems.
Delay or error between the assumed laser light transmission and the actual laser light transmission may cause an error in the detected distance to an object.
However, the phase difference between the emitted and reflected light may not align with the ideal situation represented by line 302, due to both cyclic and offset errors. For example, line 304 may represent a measured phase difference which includes offset error 306 and cyclic error 308. Offset error 306 is a constant phase error between the ideal and measured phase differences, resulting in a constant distance error when the distance to the object under measurement is calculated. Cyclic error, for example the variance shown by reference numeral 308, results in a varying or oscillating error on the detected phase measurement.
Cyclic errors may be introduced through harmonics in the laser modulation. For example, some systems may use square-waves to drive the laser, due to the lower complexity and cost of generating square waves compared to the generation of sinusoidal waves. A perfect sinusoid includes a frequency component only at the fundamental frequency, whereas square waves also include further harmonic frequency components. Other sources of harmonics can be the VCSEL response to the electrical signal driving it, or non linearities in the pixels demodulating characteristics. Removing cyclic errors introduced by square-wave laser driving can allow the system to use a square wave drive signal instead of a sinusoid, reducing the computational and power cost of generating the drive signals. Driving with a square wave also for a given maximum current creates a larger fundamental frequency component than a sine wave of the same magnitude, but at the expense of harmonics which contribute to the cyclic errors.
The offset error may be introduced by fixed delays in the timing of drive/control signals within the ToF system. In one example, there may be a delay between the timing of the laser drive signal and timing of when light is actually emitted from the laser, for example by virtue of the time it takes for the modulated laser drive signal to reach the laser driver 105, be processed by the laser driver 105, reach the laser 110 and then excite light to be emitted from the laser 110. Some systems may have more than one laser with more than one driver and control signal, which may vary independently.
This is often done to increase the illumination power which helps the TOF system work over wider angles and/or distances To provide accurate distance measurement, both the cyclic error and the offset error may be reduced or removed during the process of determining a depth frame/image using the data readout from the imaging sensor. This first requires calibration of the ToF system to determine the cyclic and offset errors. Calibration is typically performed on a device-by-device basis in the factory, but this is this is both costly—requiring further manufacturing time—and challenging. Further, the delays or error in the ToF system may drift over time or over temperature, resulting in a system whose accuracy reduces or varies over time or temperature. The effect with temperature may be reduced by having a learnt change in response with temperature, and a measurement of temperature and combining them to perform a correction. For practical reasons the change in response with temperature is characterised on a number of units and then applied to all, to avoid per unit cross temperature testing, but here the variation between a given unit and the characterised unit with temperature will also lead to an error.
It is therefore required to provide an improved system of reducing or removing these errors in the phase measurement.
The system 400 further includes a secondary/second array or imaging sensor 470. The secondary array 470 is positioned such that there is an optical path 480 between the light emission unit 410 and the secondary array 470, the optical path 480 being internal to the system 400. The optical path 480 is a known, constant, distance, such that light emitted by the laser 410 travels over a known distance before it is received by the secondary array 470. The optical path 480 may, for example, comprise fibre optics and/or one or more mirrors and/or one or more prisms, and is configured such that a part (typically a relatively small part) of the light emitted from the light emission unit 410 enters the optical path 480 and is guided to the secondary array 470. The optical path may be largely through free space, or through material that partially attenuates such as moulded black but still allows some part of the light to reach the secondary array. The remainder (typically a relatively large part) of the light emitted from the light emission unit 410 forms the emitted light 450 This emitted light may go through a diffuser array or other optical component such as dot projector or diffractive optics, mirror and/or refractive lens on the way to the scene/object 440.
The light fed to the local optical path 480 may be taken from reflected or scattered light from any of these elements. The relative optical light power in the path 480 is proportional to the optical power of the emitted light 450 and largely static in relationship. The effective distance of the coupled light from the laser to the secondary array along the optical path 480 can be considered largely constant although temperature or drift may change it very slightly. In cases where there are multiple light emission units 410, multiple optical paths 480 could be used to feedback to one or more secondary arrays 470.
Whilst the system shown in
As the secondary array 470 is a known optical distance from the light emission unit 410, the theoretical phase shift that should occur between the emitted light and the received light is a known quantity which may be calculated. However, due to offset and cyclic errors, the actual measured phase difference between the emitted light and the light received at the secondary array 470 will not be the same as this theoretical phase shift. Therefore, by measuring the phase shift which occurs between the emitted light and the light received at the secondary array 470, and comparing this measured value to the previously calculated theoretical value, an offset and cyclic error for the system may be calculated.
For example, the phase shift between the emitted light and light received at the secondary array 470 may be calculated using the method described with respect to
As the cyclic error 308 varies over the ideal/inserted phase difference (equivalent to a variance over distance travelled), the system 400 may need to acquire charge samples across the phase range. This is achieved by varying the relative timing of the accumulation period of the secondary array 470 and the modulated control signal 490, such that samples are taken by the secondary array 470 at different phase offsets, for example, at 0°, 90°, 180° and 270°, as shown in
Taking a greater number of charge samples at different phase relationships compared to the fundamental frequency of the modulated control signal 490 allows the detection of cyclic errors caused by a greater number of harmonics, introduced by, for example, square wave laser driving. By detecting the cyclic errors more accurately, it may be possible to reduce them more effectively with compensation. To acquire a number of different charge samples each at different phase offsets as described earlier with reference to
Once the system 400 offset and cyclic errors are determined using the data readout from the secondary array 470, compensation may be applied (as explained in more detail below) so as to reduce the errors and improve the accuracy of depth frames determined using the main array 420.
The internal optical feedback path 480 between the light source 410 and the secondary array 470 should have a known optical distance. To achieve this, a form of optical guide between the laser 410 and the secondary array 470 may be provided. For example, an optical fiber, mirror, lens or refractive/diffractive element may be positioned between the laser 410 and the secondary array 470 to guide to the secondary array 470 a portion of the light emitted from the laser 410.
In some instances, light 480 which is routed to the secondary array 470 may also be detected at the main array 420 due to scattering or spreading of the light 480. This light is not reflected from objects, and therefore provides no useful information to the main array 420 relating to distance/depth determination of objects and may instead reduce the accuracy of distance/depth determinations. It may therefore be beneficial to reduce the reception of light 480 at the main array 420. With this in mind, optionally a blocking element 530 may be positioned between the main array 420 and the secondary array 470 to reduce the proportion of light 480 directed at the secondary array 470 which reaches the main array 420. Where the main array 420 and the secondary array 470 are co-located on the same chip, the main and secondary arrays may be located on different parts of the chip so that there is space to position the blocking element 530 between them. It will be appreciated that the blocking element 530 is optional and may be omitted entirely, for example where the physical arrangement of the main and secondary arrays and/or the optical guide 520 are such that there is no problem, or potential for, any part of the light 480 being incident on the secondary array 470.
Further, the light 480 directed at the secondary array 470 may be of a lower intensity than the light 450 used for imaging the environment. For example, the optical guide 520 may be positioned so that the portion of light 480 routed to the secondary array 470 is at least an order of magnitude smaller than the light 450 directed into the wider environment for distance or object detection. For example, the portion of light 480 routed to the secondary array 470 may be 1/10th, 1/100th, 1/1000th, 1/10000th or less than 1/100000th of the total light produced by the light source 410, such that even if some of the portion of light 480 scatters or spreads and reaches the main array 420, it is likely to be of a significantly lower intensity than the light 460 which has reflected off objects in the environment. Furthermore, by redirecting only a relatively small amount of the emitted light to the secondary array 470, the intensity of light 450 can be kept high, which may be of benefit for ToF imaging.
Whilst
A secondary array or secondary image sensor 470 arranged to receive light that has been internally fed-back may optionally provide further features to a ToF system which would not otherwise be possible. In these situations, the secondary array 470 may provide the additional features as well as or instead of determination of the offset and/or cyclic errors.
ToF systems are often used in safety-critical applications. For example, in vehicles, ToF may be used to provide an auto-brake function when a pedestrian is detected, or to prevent collisions with other vehicles. If the ToF system malfunctions, and there is no or limited knowledge that the system has malfunctioned, decisions may be made based on erroneous distance measurements. This may result in safety-critical decisions being made on the basis of erroneous data.
The secondary array 470 of the present system may optionally be useful for monitoring further characteristics of the light source 410, such as the phase response, amplitude, luminance, light intensity and/or the linearity of the laser response. For example, the controller may be configured to monitor the amount or intensity of light received at the secondary array 470 in order to monitor the performance of the light source 410. The measured response of the system while operating can be compared with one of a batch characterised response of a number of similar units, an expected response from modelling or simulation, or a measured response of the unit when it was initially manufactured and tested, or a definition in one or more of the parameters of offset distortion. Intensity or output power above or below an acceptable amount, for example intensity which is greater than a threshold or less than a threshold, can be used to flag a pending or actual issue in the system.
During object detection using the main array 420, the secondary array 470 should receive a known amount of light from the laser 410 along optical path 520. If the amount of light received at the secondary array 470 is outside of normal range, the system may halt or stop operation of the ToF system, or provide reduced assurances as to the accuracy of the distance measured by the ToF system (for example by flagging that any output depth/distance frames may have a reduced accuracy). This may ensure that the level of laser light transmitted is within safe bounds. Too much light emitted by the laser 410 may damage the eyes of people within range, whereas too little may result in objects being undetected, or detected with lower accuracy, due to the lower illumination of the scene. Further, in a situation where the light directed at the secondary array 480 is guided using an optical waveguide 520 which is integrated with the diffuser 510, too little light may indicate that the diffuser 510 has become displaced which is an eye safety issue. Too much light could also indicate a change such as the laser driver power supply had increased or the diffuser was displaced in a way such as to couple more light, but a significant change would still be indicative of a likely failure.
The system may compare the measured characteristic of light with a predetermined operating range or threshold range. As described above, the threshold range may relate to acceptable operating parameters outside of which the system does not operate in a satisfactory manner. For example, the threshold range may define a range in which eye damage does not occur, but which still allows accurate detection of objects. Based on this, the secondary array 470 may provide feedback to a diagnostic or safety system as to whether the system should shut down or alter operation.
The system may monitor the absolute light intensity received at the secondary 470 image sensor. Alternatively, the system may monitor the intensity of light received over a period of time. Where there is a deviation in the intensity of the received light outside of the normal intensity of received light, the system may activate a safety function of the ToF system, for example turning off the ToF system. The normal range of operation of the light source 410 may be determined on a device-by-device basis during factory calibration, with numerical limits being stored in memory in the system (not represented in the figures) for use during operation of the ToF system. The relationship of the expected proportion of light coupled to the secondary array 470 relative to the amount of light emitted to the scene 410 may be factored in. A normal range of operation may be known from the specification of the light source 410, design of the optical guide 520 and the secondary array 470, such that a predetermined normal range of operation may be stored in memory without requiring device by device calibration.
In industrial settings and applications, ToF systems may be operated using multiple light sources to increase the illumination available, providing improved accuracy of depth determination. In some systems, the light sources may be operated with the same control signals, such that the lasers are operated in sync.
However, differences in light sources themselves, and/or the control paths and circuitry of the first and second light sources, and/or other PVT errors, may cause the light output from the two light sources to differ. For example, the light they emit may have different offset and/or cyclic errors. Imaging the same scene with two lasers with different offset and cyclic errors may result in an error in the distance measurements which in the past has been very difficult to correct as the contribution of each light source cannot easily be separated.
The internal optical feedback arrangements of the present disclosure may be used to address this issue. In particular, the process described above with respect to determining the offset and cyclic errors of a single light source may be applied to a system with two or more light sources, such that the relative offset and cyclic errors of each light source are determined. For example, optical guides maybe used to internally feedback a portion of the light emitted from light source 1610 to the secondary array 470 and internally feedback a portion of the light emitted from light source 2620 to the secondary array 470. During error determination, the controller 430 may be configured to operate just light source 1610, then determine offset and/or cyclic errors of that light source using the secondary array 470. Subsequently, the controller 430 may operate just light source 2620, then determine the offset and/or cyclic errors of that light source using the secondary array 470. In an alternative, a separate secondary array may be provided for each light source, such that both light sources may be operated concurrently during error determination, which may enable error determination to take place at the same time as depth/distance determination using the main array 420. Optionally, the optical feedback path for each light source may be completely different, or at least part of the optical feedback path for light emitted from the first and second light sources may be shared.
The system 600 may include a variable or controllable delay 650 in the control signal path of light source 2620. Once the offset errors of the two light sources have been determined as explained above, the controller 430 may set the time delay applied by the delay block 650 such that relative offset errors between light emitted from the two light sources is reduced or removed. In an alternative, the delay unit 650 may be positioned to delay modulation signal 630, or a delay unit may be positioned in the modulation signal path for each light source. In the latter example, the controller 430 may be configured to achieve the same, predetermined overall offset for both light sources, such that once the offset error of each light source has been determined, an appropriate delay is added to the modulation signal for each light source so that both light sources have the same total offset. Instead of using a delay 620 in the path of one of the light sources, alternative means for calibrating the first 610 and second 620 light sources may be used to insert an effective phase delay.
Whilst
Secondary Array with Optical Control
Optionally, an optical control switch 720 may be included in the internal optical path between the light source 410 and the secondary array 470.
In some implementations, light which is internally fed-back to the secondary array 470 may inadvertently be detected by the main array 420 due to scattering or spreading of the light. In this case the optical switch 720 may be configurable such that the light path between the laser 410 and the secondary array 470 to be turned off when not in use for any specific function, for example, when not being used to determine safe operation of the laser, or when not being used to determine offset and cyclic errors.
Offset and cyclic errors may be measured and calculated intermittently or periodically, for example, hourly, daily, weekly or monthly, or every time the ToF system starts up. When the offset and cyclic errors are not being measured, the optical switch 720 may be engaged to prevent light being internally fed back to the secondary array 470, thereby reducing the potential for light to scatter and be detected at the main array 420. This reduces the incidence of multipath light detection by the main array 420, ensuring that each object is illuminated solely, or principally, by the main light path 450 when the secondary array 470 is not in use. Furthermore, when there are two or more light sources, as described above with reference to
Periodically, the main array 420 may be used to determine the cyclic and offset errors present in the ToF system 800 using light directed through optical guide 710. During error determination, external light 460 may be blocked from illuminating the main array 420 (for example, by virtue of a controllable shutter, not represented in the figure) and the optical switch 720 may be controlled to allow the light 480 to pass to the main array 420. When the system 800 is operating in normal distance/depth determination mode, external light 460 may be allowed to illuminate the main array 420 and the optical switch 720 may be controlled to prevent light travelling down the optical guide 710. With the use of such periodic or intermittent optical control, the main array 420 may at times be used for offset/cyclic error determination, safety functions and/or or calibration of multiple lasers, as described in the various examples above.
Where the main array 420 is used to provide the function of the secondary array 470, a sub-part of the main array may be used. For example, the optical guide 710 may direct light along an optical path onto only a part of the main array 420. Alternatively, internally fed-back light 480 may illuminate the whole of the main array 420.
Using the main array 420 to perform the secondary array functions described above may help to reduce the size of the ToF system, as a secondary array is not required.
To determine the offset and cyclic error of the system, a number of samples are taken using the secondary image sensor at different phases, by adjusting the relative phase of the light source modulation signal and the accumulation control signal of the secondary array (also referred to as the demodulation signal used to control the secondary array). When determining cyclic error, a greater number of samples may increase the accuracy of the determination, for example by detecting the effects of higher order harmonics. However, an increased number of samples takes a longer period of time when compared to taking a smaller number of samples. There is therefore a trade-off between the time taken to determine the cyclic error and the accuracy of the determination. Faster acquisition can be useful in systems that are used periodically. In periodic systems, changes in the ambient conditions in between uses mean that the offset and cyclic errors may have changed between the last time of operation and the current time of operation. Accurate operation is important from as soon as the system is used, and therefore fast acquisition of cyclic errors is important. Faster acquisition can also help track changes.
It may be possible to increase the number of different phase samples from the secondary array 470 compared with the main array 420 by controlling the two arrays with different accumulation control signals. However, this may increase the complexity of control, since the controller 430 would need to be configured to output two independent array control signals with different amounts of phase offset relative to the modulated light signal. Furthermore, increasing the number of different phase samples acquired by the secondary array 470 will further increase the time taken to perform offset and cyclic error determination.
In view of this, the inventors have developed an implementation that includes a plurality of secondary imaging sensors/arrays (e.g., two or more secondary imaging sensors/arrays).
Each internal feedback array 920, 930, 940, 950 may be controlled using a separate accumulation control signal, such that the timing of accumulation for each array may be different, or skewed, relative to each other. Put another way, each internal feedback array may be controlled to “open its shutter” for charge accumulation at different relative phases, for a different phase range, of the light modulation signal. However, a single accumulation control signal may still be output by the controller 430 for controlling the plurality of secondary arrays 910 (which may also be the same accumulation control signal used for the main array 420). In this case, each internal feedback array 920-950 may be configured to apply a different delay to the received accumulation control signal, such that the actual accumulation timing of each internal feedback array 920-950 may be different to each other.
The third internal feedback array 940 may apply a different delay to the accumulation control signal, for example one that equates to a 30° phase delay. In this example, during the first accumulation period 10101, the third internal feedback array 940 would accumulate charge between 30°-210° of the light modulation signal. The fourth internal feedback array 950 may apply a further different delay to the accumulation control signal, for example one that equates to a 45° phase delay. In this example, during the first accumulation period 10101, the fourth internal feedback array 950 would accumulate charge between 45°-235° of the light modulation signal.
Another way to generate the relative phases is to have multiple control lines coming from the controller 430 that might output the control lines from multiple stages within a DLL or PLL that may already be present for generating the relative phase between the control signal for the main array and the light source.
The relative delays do not need to be evenly spread—as long as they are relatively well known they can be accommodated for in the calculation of offset and cyclic error.
The plurality of second arrays may be implemented as different columns of pixels abutting each other but which are driven by the different control signals. The secondary light (480) being feedback may have a slightly different path length to the different sub arrays and this may also be accommodated for in the calculation. There can also be multiple sub arrays at nominally the same delay, that are inter digitated in some way as to minimise this source of error.
By operating the plurality of secondary arrays 910 in this manner, the phase of light received at the secondary arrays 910 relative to the light emitted from the light source 410 over the intervals noted above may be determined using the charge accumulated on the plurality of secondary arrays 910, for example by cross correlating the accumulated charge signals with the modulation signal 490. In this example, each accumulation interval for the plurality of secondary arrays 910 spans half of the period/cycle of the light source modulation signal 490, but may alternatively take place for any other suitable amount of time, for example for one quarter or one eighth of the period of the light source modulation signal 490. As will be understood by the skilled person, if the plurality of secondary arrays 910 are of a single ended pixel type, the pixels may be controlled to accumulate charge for this part/interval of the period and not accumulate any charge for the remainder of the period. If on the other hand they are of a differential pixel type, the pixels may be controlled to accumulate charge for this part/interval of the period on one side of the pixel and accumulate charge on the other side of the pixel for the remainder of the period.
During the read-out period of time 10201, controller 430 may control the light source 410 to cease emitting light and readout from the plurality of secondary arrays 910 image sensor values that are indicative of the charge accumulated in the imaging pixels of each internal feedback array 920-950. The nature of the readout values will depend on the technology of the internal feedback arrays 920-950. For example, if they comprise CMOS sensors, voltage values may be readout, where each voltage value is dependent on the charge accumulated in an imaging pixel, such that the readout voltage values are each indicative of charge accumulated in imaging pixels of the plurality of secondary arrays 910. In other sensor technologies, the nature of the readout values may be different, for example charge may be directly readout, or current, etc. For example, the plurality of secondary arrays 910 may be controlled to readout image sensor values row-by-row using any standard readout process and circuitry well understood by the skilled person. In this way, a sample of charge accumulated by each imaging pixel during the accumulation period 10101 may be read off the plurality of secondary arrays 910, converted to a digital value and then stored by the controller 430. The group of values, or data points, arrived at the conclusion of this process is referred to in this disclosure as a charge sample, such that at the end of readout period 10201, the controller 430 will have obtained a plurality of charge samples, one for each of the plurality of secondary arrays 910.
It will be appreciated that the accumulation period of time 10101 may last for multiple periods/cycles of the first modulation signal (as can be seen in
During the second accumulation period of time 10102, the controller 430 controls the first source 410 to output light modulated by the modulation signal 490 for an accumulation period of time 10102. This is very similar to the accumulation period 10101, except during accumulation period of time 10102 the accumulation control signal output by the controller 430 to the plurality of secondary arrays 910 (i.e., the demodulation signal) is of a different phase relative to the modulation signal 490. In this particular example, it is 90° out of phase with the modulation signal 490 such that each of the internal feedback arrays 920-950 accumulate charge for a different part/interval of the period/cycle of the modulation signal 490 (90°-270°, 105°-285°, 120°-300°, 135°-315°, respectively, as can be seen in
Third accumulation period of time 10103 is very similar to the period 10102, except the accumulation control signal output by the controller 430 is again of a different phase relative to the modulation signal 490. In this particular example, it is 180° out of phase relative to the modulation signal 490 such that each of the internal feedback arrays 920-950 accumulate charge for a different part/interval of the period/cycle of the modulation signal 490 (180°-360°, 195°-15°, 210°-30° or 225°-45°, respectively, as can be seen in
Finally, fourth accumulation period of time 10104 is very similar to the period 10103, except the accumulation control signal output by the controller 430 is again of a different phase relative to the modulation signal 490. In this particular example, it is 270° out of phase relative to the modulation signal 490 such that each of the internal feedback arrays 920-950 accumulate charge for a different part/interval of the period/cycle of the modulation signal 490 (270°-90°, 285°-105°, 300°-120°, 315°-135°, respectively, as can be seen in
It can be seen from the above that for each accumulation period 10101-10104, the start timing of pixel accumulation timing relative to the light modulation signal 490 is shifted (i.e., the relative phase of the light modulation signal 490 and the pixel demodulation signal, which controls pixel accumulation timing, is shifted) for each of the plurality of secondary arrays 910. In this example, it is achieved by the controller 430 adjusting the relative phase of the accumulation control signal that it outputs to the plurality of secondary arrays 910 for each accumulation period 10101-10104, with each of the internal feedback arrays 920-950 then applying a different delay to that signal. The light modulation signal 490 is unchanged for each accumulation period 10101-10104. As noted previously, the controller 430 may instead output a plurality of control signals to control the accumulation periods of the plurality of secondary arrays 910 individually.
However, as an alternative the relative phase of the light modulation signal 490 and the accumulation control signal (e.g., the demodulation signal) may be changed for each accumulation period 10101-10104 by adjusting the light modulation signal 490 and keeping the accumulation control signal fixed. For example,
Whilst in these examples each accumulation period 10101-10104 lasts for 50% of the period of the laser modulation signal (i.e., for 180°), in an alternative each accumulation period may be shorter, for example 60°, or 90°, or 120°, etc, with the start of each accumulation period for one of the image sensors relatively offset by 90° as explained above. Each of the image sensors may have an offset accumulation period relative to one another other.
After completing the four accumulation periods 10101-10104, 16 samples of data (charge samples) have been acquired and stored in memory, four samples from each of the internal feedback arrays 920-950. They together may be referred to as a first set of charge samples. Immediately after the read-out period 10204, or at some later time, a phase relationship between the light emitted from the light source 410 and the internally fed-back light 480 may be determined using the 16 charge samples (for example by performing a discrete Fourier transform (DFT) on the samples to find the real and imaginary parts of the fundamental frequency, and then determining the phase from the real and imaginary parts, as will be well understood by the skilled person). There is noise associated with each measurement, however the more pixels that are inside each array the greater the reduction in noise can be. Further, many repeated measurements can be made with the same array to also reduce noise.
This measured phase relationship may be compared to an expected/ideal phase relationship for the internally fed-back light 480 (e.g., the phase relationship that would be expected for the distance the internally fed-back light 480 travels, if there were no offset or cyclic errors). In some instances, the internal optical feedback path between the light source 410 and the plurality of secondary arrays 910 is so small that, given the speed of light, the expected phase relationship may be that the light arriving at the plurality of secondary arrays 910 is perfectly in phase with the light emitted from the light source 410. By comparing the measured phase relationship with the expected/ideal phase relationship, offset and cyclic errors of the system may be determined.
Whilst in this example the plurality of secondary arrays 910 includes four imaging sensors, it is clear that it may comprise any number for example, 2, 3, 4, 5, 6, 7, 8, 10, 12, 16, 24, 32 etc. Further, whilst four accumulation periods are shown in
A two-tap demodulating pixel is a pixel that accumulates charge in each of its respective taps. The demodulating pixel is supplied with two control signals (for example control signal phase-a and control signal phase-b). During a high period of control signal phase-a (or however the control signal represents that accumulation should take place), the pixel will accumulate charge on the first tap. During a high period of control signal phase-b, the pixel will accumulate charge on the second tap.
It should be noted that whilst
The scheme shown here means that as the different light phases are used versus the primary array the plurality of secondary arrays builds up a total of 12 points from 0/180deg to 165/345 degrees, 4 points at a time, and in the any period 1110x there are 4 points available to use that span the 180/360 degrees. Each of the secondary arrays are spaced equally in time by delays equivalent to 45degs. They do not need to be spaced equally as long as how much they are spaced is understood to the required tolerance, this could for example be measured and accounted for at test.
A calculation of offset can be done with just one illumination, and a better calculation of offset and a build-up of cyclic error can be done with the illuminations that would normally be used for the light source versus the primary array so no adjustment of the multiple secondary array phases versus the light source is needed to be able to perform any extraction of offset or cyclic error of this fixed path and use that to modify the light source relative delay and/or its shape, or to be used in any subsequent calibration or correction of depth measured by the primary array.
By accumulating charge over three periods 11101-11103 the system may arrive at a determination of cyclic error faster than the system of
Note if the distance of the optical path from the light source to the plurality of secondary arrays is different this may also be accounted for in the calculation of offset or cyclic error.
Many other schemes with multiple secondary arrays are possible.
The use of multiple secondary arrays to determine at least one of the offset or cyclic error may allow the ToF system to be calibrated or corrected to reduce the impact of the offset or cyclic error. Once offset or cyclic error has been determined, ToF distance measurements performed using the main image sensor may be calibrated or corrected to remove the errors. For example, the controller may be configured to use the determined error of the ToF system to modify the light outputted by the light source to reduce the error of ToF system. This would reduce the error in future distance measurements performed using the ToF system.
Alternatively, the system may correct the depth map or distance measurement performed using the main image sensor after the depth map or distance measurement has been performed. For example, the controller may be configured to correct the ToF distance measurement digitally using the determined offset and cyclic error. For example, the phase relationship shown in
Correction/Reduction of Offset and/or Cyclic Errors
The offset and cyclic errors present in the ToF system may be determined in any number of ways, as described above. For example, one or a plurality of secondary arrays may be used to accumulate charge from incident light which travels down an internal optical feedback path of a known length, and the phase of the detected light compared to a theoretical phase shift which should occur over the known path. Alternatively, the main, or primary image sensor may be used in conjunction with an optical switch. Once the magnitude of the offset and cyclic errors are determined using any of the previously described methods and systems, a calibration may be applied to the ToF system to remove or reduce the effects of the error.
In some situations, the error may be reduced or removed performing post-processing determination of the depth frame using data readout from the main array 420. For example,
During operation of the system 1300, the main array 420 may accumulate charge in a number of accumulation periods, as described earlier with reference to
Post-processing calibration may be applied by the error correction unit 1340 to the phase calculations determined by the phase calculation unit 1320. Phase unwrapping 1350 may then be performed on the corrected phase calculations in order to determine the distance to the imaged object (i.e., determine the depth frame A point cloud, or depth frame, may be generated by the point cloud generation unit 1360 to provide a map of the measured distances. The point cloud, or depth/distance frame, may be provided 1370 to external applications or third-party devices for use.
The inventors have recognised that correcting the offset and/or cyclic errors by post-processing may increase the computational intensity of determining the depth frame. To address this, the inventors have developed techniques for modifying the light modulation signal used to drive the light source 410 in order to reduce or remove the cyclic and/or offset errors.
Error detection is by error detection unit 1330 using charge accumulated on the secondary image sensor 470, as explained earlier. However, rather than using the determined error to correct the output of the phase calculation unit 1320, the system may instead modify the modulated signal used to control the light source 110. There are various different ways of modifying the modulation signal, any one or more of which may be used.
The modulation signal generator 1410 may pre-distort the modulation signal in order to change the shape of the signal. Changing the shape of the signal should change the harmonic content of the signal. The harmonic content of the signal contributes to the cyclic errors of the ToF system, so the modulation signal is pre-distorted in order to change its harmonic content in a way that reduces or removes the cyclic errors.
Pre-distortion may be performed by the modulation signal generator 1410 in one or more different ways. For example, predistortion may be applied such that a different predistortion is applied to different frequencies of the modulation signal, as the harmonic signature varies with frequency. The pre-distortion can also modify the effective delay of the fundamental of the light if required. In one example, it may change the duty cycle of the modulation signal. For example, modulation signal may be a square way signal having a 50:50 duty cycle, meaning that the signal is ‘high’ for 50% of the signal period and is ‘low’ for the other 50% of the signal period. The modulation signal may be predistorted by changing that duty cycle, for example to 60:40. Additionally or alternatively, the modulation signal generator 1410 may change the shape of the modulation signal in any other suitable way, for example modifying the amplitude of the modulation signal, introducing additional amplitude levels to the signal so that it more closely replicates a sine wave signal, or more closely replicates a signal such that after the laser driver and through the laser the light contains less of the harmonics, changing the magnitude different amplitude levels and/or duration spent at each amplitude level for a multi-level signal, etc. Although we show the modulation signal generator making the change, depending on the partition and capability of the components it is also possible to do it elsewhere, for example additionally or alternatively the system may communicate to the laser driver which may contain a means to alter the pre-emphasis of the drive or may contain a DAC and a PLL and be able to create multiple levels within one period to alter the shape as requested.
Additionally or alternatively, the modulation signal generator may change the modulation signal partway through the accumulation periods 2101-2104, 2301-2304 (which are described earlier with reference to
ToF systems may operate at multiple frequencies. For example, a continuous wave ToF system may operate at a first frequency, then at a second frequency, comparing the phase relationships at these different frequencies so as to perform phase unwrapping. Due to the nature of harmonics, at least one of the cyclic and offset errors may be different at these different frequencies. As such, the system may operate to determine the cyclic and offset error at a first frequency, then determine the cyclic and offset error at a second frequency. The system may then apply different error corrections depending on whether the ToF system is operating at the first frequency of the second frequency. Whilst frequency correction is described here with respect to two frequencies, the system may operate in a similar manner to correct more than two frequencies, for example, three, four, or six frequencies.
Because of the integrating nature of the main array 420 during an accumulation period, if the modulation signal generator 1410 outputs signal 1 as the modulation signal for a first part of the accumulation period (for example, the first half) and then outputs signal 2 as the modulation signal for a second part of the accumulation period (for example, the second half), this effectively combining the two signals, the result of which is represented in
The combination of the two signals in this example may approximate/synthesise a sine signal having the same fundamental frequency as signal 1 and signal 2. As can be seen, the combined signal approximates/synthesises a sine signal by oscillating between two energy levels (e.g., between two extreme energy levels, or max and min) and at least one intermediate energy level (e.g., between the two extreme energy levels), such that the combined signal is closer in nature to a sinusoidal signal compared with the two square wave signals Signal 1 and Signal 2. As a result, by combining two different square wave signals in this way, a sinusoidal illumination signal may be approximated such that the amplitude of at least some of the harmonic content in the combined signal is lower than the amplitude of the corresponding harmonic content in the first and second signals. This may reduce or remove the cyclic errors of the system. The charge accumulation timing described earlier with reference to
The harmonic content of the combined signal may be adjusted by changing the amount of time each of signals 1 and 2 are used as the modulation signal; and/or by changing the duty cycle of one or more of signal 1 and signal 2; and/or by changing the amplitude of one or more of signal 1 and signal 2. Therefore, pre-distorting the modulation signal in this way may be used to reduce or remove the cyclic errors of the system. Whilst in this example each different signal consecutively, one at a time, during the accumulation period, in an alternative the modulation signal generator 1410 may be configured to generate and combine the two or more signals itself, so that the modulation signal output to the delay unit 1420 is the combined signal.
In the above, various different examples are disclosed for how the modulation signal generator 1410 may pre-distort the modulation signal in order to reduce or remove system errors determined by the error detection unit 1330. Additionally or alternatively, the modulation signal may be pre-distorted by adjusting at least one setting of the driver 1430. The light source driver 1430 is configured to receive the modulation signal and generate a drive signal based on the modulation signal (for example, the drive signal may be an amplified version of the modulation signal), the drive signal being used to drive the light source 410 to emit light. As will be well known to the skilled person, such light source drivers 1430 typically have one or more adjustable settings that define how the light source driver 1430 responds to the received modulation signal, for example how it responds to the rising and falling edges of a square wave signal. For example, the adjustable settings may comprise one or more of: a sag compensation setting; an over/under shoot setting; or to synthesize a arbitrary waveform for example by having a DAC that uses a clock locked to the incoming signal to generate adjustable steps at a higher frequency. The laser driver could also have one or more of its analog characteristics altered, for example any filtering or resonance could be adjusted. Adjusting one or more or these settings should change the shape of the drive signal output in response to the modulation control signal, which should change the harmonic content of the light emitted from the light source 410, which should in turn change the cyclic error of the system. The signal 1435 output by the error detection unit 1330 may comprise the one or more driver settings, which the error detection 1330 may determine based on the measured system errors in order to reduce the system errors. Alternatively, the signal 1435 may comprise the measured system errors, which the driver 1430 may use to determine the adjustment to the one or more driver settings in order to reduce the system errors.
In a further example represented in
Whilst
Each of the disclosed modifications to the modulation signal may be performed in a number of different ways. For example, the required modification may be calculated based on the determined ToF system errors, or it may be looked up in memory where suitable modifications for different levels of system error may be stored (which may have previously been determined, for example, experimentally). Additionally or alternatively. The modifications may be performed in an iterative manner. For example, a modification may be made and the offset and/or cyclic error may be measured. The modification may then be iteratively changed and the offset and/or cyclic error remeasured until the error has reduce to an acceptable level or been eliminated entirely. The delay and modulation modifications may be different for all the frequencies being used in the system, but the system may use information from one to adjust the starting point for the other, as some characteristics may be translatable.
Also, while the system could be driving down the measurement of cyclic error to minimise it based on the number of points of phase that have been measured, and the harmonics being measured it may by taking into account a model of the light emission path and be able to drive down harmonic content in higher harmonics using that model. For example, a model of the light emission chain could be characterised to a higher number of sampling points over a number of different units, and a machine learnt model created that takes in the lower number of measured points in a given unit but predicts a correction that covers higher order errors with an improved probability based on the learnt relationship.
The skilled person will readily appreciate that various alterations or modifications may be made to the above described aspects of the disclosure without departing from the scope of the disclosure.
