Patent Application
20230011969
The disclosure relates to a method of time of flight sensing, and to a time of flight sensor system.
The present disclosure relates to a method of time of flight sensing. Time of flight sensing determines the distance of an object from a sensor using the known speed of light. In one example a pulse of light (e.g. at an infra-red wavelength) is emitted from a light source, and light reflected back towards the sensor from an object is detected. The source and the sensor may be located adjacent to one another (e.g. as part of the same integrated system). The distance between the sensor and the object is determined based upon the elapsed time between emission of the pulse of light and detection of the pulse of light by the sensor.
The time of flight sensor may be an imaging array which comprises an array of pixels. The pixels may for example be single photon avalanche photodiodes (or some other form of photo-detector). The imaging array may provide a ‘depth map’ which indicates the measured distance of objects from the sensor in the form of an image.
The intensity of light which is received by a time of flight sensor following reflection from an object reduces as a function of the square of the distance between the object and the sensor. As a result, the signal to noise ratio at the sensor when light is reflected from a distant object (e.g. at a distance of 5 metres or more) may be low. In one known time of flight sensing method, in order to improve the signal to noise ratio for distant objects, the signals output from multiple pixels of the sensor are combined together. The multiple pixels may be arranged as a square or rectangle, and may be referred to as a macropixel. Because the macropixel has a larger surface area than a single pixel it receives more photons of light, and thereby provides an output which has a better signal to noise ratio. However, grouping pixels together in this way is disadvantageous because it reduces the resolution of the depth map which is obtained.
In one known method this problem is addressed by obtaining one depth map from the time of flight sensor at a low resolution (using macropixels) and in addition obtaining a further depth map from the time of flight sensor at a high resolution (using individual pixels). These two depth maps are then added together by a processor. The resulting combined depth map consists of high resolution areas where close objects are present and low resolution areas where distant objects are present.
A disadvantage associated with the above approach is that in order to obtain the two depth maps the light source must be operated two times. This uses a significant amount of power. This is particularly disadvantageous if the sensor forms part of a hand held device, e.g. a mobile phone, because battery life of the device is of key importance to the user and operation of the light source will deplete the battery.
It is therefore an aim of the present disclosure to provide a time of flight sensing method that addresses one or more of the problems above or at least provides a useful alternative.
In general, this disclosure proposes to overcome the above problems by operating groups of photo-detectors in a first mode in which the outputs from the photo-detectors are combined together, whilst at the same time operating other groups of photo-detectors in a second mode in which the outputs from the photo-detectors are not combined together. The first mode may be used for distant objects and the second mode may be used for close objects. This arrangement advantageously provides low resolution depth mapping for distant objects (whilst providing an acceptable signal to noise ratio), and at the same time provides high resolution depth mapping for close objects. This is advantageous because it avoids the prior art requirement to obtain a full low resolution depth map and subsequently obtain a full high resolution depth map. Obtaining two full depth maps according to the prior art requires the emission of more light pulses from a light source, and thus causes faster battery depletion.
According to one aspect of the present disclosure, there is provided a method of time of flight sensing, the method comprising using an emitter to emit pulses of radiation, using an array of photo-detectors to detect radiation reflected from an object, for a given group of photo-detectors of the array, determining based upon measured times of flight of the radiation, whether to use a first mode of operation in which outputs from individual photo-detectors of the group are combined together or to use a second mode of operation in which outputs from individual photo-detectors are processed separately, wherein the array of photo-detectors comprises a plurality of groups of photo-detectors, and wherein one or more groups of photo-detectors operate in the first mode whilst in parallel one or more groups of photo-detectors operate in the second mode.
Thus, embodiments of this disclosure advantageously allow distance data for different resolutions to be obtained in parallel from different parts of the same photo-detector array.
When the method beings, groups of photo-detectors may initially operate in the first mode of operation.
When the method begins all groups of photo-detectors may initially operate in the first mode of operation.
The method may switch from the first mode of operation to the second mode of operation for a group of photo-detectors if measured times of flight for that group of photo-detectors indicate the presence of an object at distance which is below a threshold distance.
The method may delay switching to the second mode of operation until sufficient measured times of flight have been received at the group of pixels in the first mode to provide a desired signal to noise ratio.
The method may switch immediately to the second mode of operation.
If measured times of flight for the individual photo-detectors do not indicate the presence of an object at the distance identified during the first mode of operation, the method may switch back to the first mode of operation.
The method may further comprise, for the given group of photo-detectors of the array, determining based upon measured times of flight of the radiation, whether to use a third mode of operation in which outputs from sub-groups of photo-detectors are combined together, the method switching from the first mode of operation to the third mode of operation for a group of photo-detectors if measured times of flight for that group of photo-detectors indicate the presence of an object at distance which is below a first threshold distance but above a second threshold distance, wherein one or more groups of photo-detectors operate in the first mode whilst in parallel one or more groups of photo-detectors operate in the third mode.
The method may further comprise one or more groups of photo-detectors operating in the second mode.
Switching between modes of operation for a given group of photodetectors may be controlled by a logic circuit which is associated with that group of photodetectors.
The logic circuit may form part of an electrical circuit which is associated with the group of photodetectors and which forms part of the same integrated circuit as the photodetectors.
The method may re-commence each time a pulse of light is emitted.
The photo-detectors may be single photon avalanche photodiodes.
According to a second aspect of the invention, there is provided a time of flight sensor system comprising an emitter configured to emit pulses of radiation, and a sensor module comprising a sensor and sensor electronics, wherein the sensor comprises an array of photo-detectors, the photo-detectors being arranged in groups, and wherein the sensor electronics comprises a plurality of electric circuits, an electric circuit being associated with each group of sensors, and wherein each electric circuit includes a logic circuit configured to determine based upon measured times of flight of the radiation, whether to use a first mode of operation in which outputs from individual photo-detectors of the group are combined together or to use a second mode of operation in which outputs from individual photo-detectors are not combined together.
The logic circuit may be configured to the electric circuit from the first mode of operation to the second mode of operation for a group of photo-detectors if measured times of flight for that group of photo-detectors indicate the presence of an object at distance which is below a threshold distance.
The logic circuit may be configured to determine based upon measured times of flight of the radiation, whether to use a third mode of operation in which outputs from sub-groups of photo-detectors are combined together, switching from the first mode of operation to the third mode of operation for a group of photo-detectors occurring if measured times of flight for that group of photo-detectors indicate the presence of an object at distance which is below a first threshold distance but above a second threshold distance.
The electrical circuit which is associated with the group of photodetectors may form part of the same integrated circuit as the photodetectors.
The electrical circuit which is associated with the group of photodetectors may further comprise a front end, a time to digital value convertor, and a memory.
The memory may be a histogram memory.
The photo-detectors may be single photon avalanche photodiodes.
Finally, the present time of flight sensing method disclosed here utilises a novel approach at least in that higher resolution and lower resolution data are captured in parallel using a single sensor.
Some embodiments of the disclosure will now be described by way of example only and with reference to the accompanying drawings, in which:
Generally speaking, the disclosure provides a method of time of flight sensing in which higher resolution data and lower resolution data are captured in parallel.
Some examples of the solution are given in the accompanying figures.
The sensor 122 is an array of photo-detectors (e.g. single photon avalanche diodes). These may be referred to as pixels. The pixels may be arranged in groups. In
The sensor electronics 110 may be provided as a plurality of independently operating electrical circuits. An electrical circuit may be associated with each group of pixels. In
One of the electrical circuits 143 is depicted in more detail on the right hand side of
The electrical circuit 143 further comprises a time to digital value convertor 114. As is depicted in
The time value is stored in a memory 116. The memory 116 may be a histogram memory. A histogram memory comprises bins representative of different elapsed times. When the photon is detected, the elapsed time value for that photon causes an increment of the bin which corresponds with the time value. Over time, many photons are detected and many associated elapsed times are measured. The data in the histogram memory represents the number of detected photons as a function of elapsed time. Peaks in the data are indicative of objects which have reflected the photons. The times associated with the peaks are indicative of the distance to those objects. The distance resolution (which may be referred to as depth resolution) provided by the histogram memory will depend upon the time duration of each bin.
The electrical circuit 143 further comprises a logic circuit 118. The logic circuit determines whether outputs from the macropixel 133 should be used individually or should be combined together (as explained further below). Logic circuits associated with different macropixels may operate independently of each other. The logic circuit may for example be a processor which performs the determination. The logic circuit may for example be a hardwired circuit which performs the determination.
The electrical circuit 143 further comprises a time to distance converter 120, which receives histogram memory outputs in the form of digital representations of time, and converts these times into distances. Outputs from the time to distance convertor 120 are passed to the image processor 106, which combines the outputs to form an image which depicts the distances to objects in the field of view of the time of flight sensor system 100. The image may be referred to as a depth map.
The pixels are grouped together, each group being referred to as a macropixel. There are nine macropixels 231-239. Each macropixel has sixteen pixels a-p (for simplicity of illustration not all of these are labelled). It will be appreciated that this is merely a schematic example, and that in practice the sensor array 222 may consist of more than nine macropixels. Similarly, macropixels may consist of more or less than sixteen pixels.
In a method according to an embodiment of the invention, five of the macropixels 231-234, 239 are operating in a first mode in which outputs of individual pixels a-p are combined together. This combining together of the outputs occurs before a time value is allocated to a detected photon. When the time value is allocated to the detected photon (which may be referred to as time-stamping), no information is recorded regarding which of the individual pixels a-p detected the photon. Thus, data stored in the histogram memory relates to the macropixel as a whole and not to an individual pixel.
Four of the macropixels 235-238 are operating in a second mode in which outputs of individual pixels a-p are processed separately and are not combined together. When a time value is allocated to a detected photon (i.e. time-stamping), the identity of the pixel which detected the photon is recorded along with the time value. Thus, data stored in the histogram memory relates to each individual pixel.
The first and second modes are operating in parallel.
The first and second modes are now described with reference to
The logic circuit 118 monitors the data held in the histogram memory to identify peaks in the data. Peaks which are identified are compared by the logic circuit 118 with a threshold time value indicative of a predetermined object distance. In an embodiment the threshold time value may correspond with an object distance of 2 m. In terms of a time value this may be expressed as 1.33×10−8 seconds (time=speed of light/distance). If a peak with a time value of less than 1.33×10−8 seconds is identified then the logic circuit may instruct that operation of the macropixel is switched to the second mode of operation. Although the threshold in this example is a time value associated with 2m, other time values may be used.
Macropixels 235-238 are schematically depicted as operating in the second mode of operation. Taking as an example macropixel 235, in the second mode of operation, the signal output from each pixel 235a-p is processed separately. Thus, when a photon is incident upon a given pixel 235a, the front end 112 causes a time value to be read out from the time to digital value convertor 114 and stored in a part of the histogram memory which is specifically allocated for that pixel. Therefore, instead of a single set of data being stored in the histogram memory, nine sets of data are stored in the histogram memory. This is relatively high resolution data. The signal to noise ratio may be sufficiently high because the object reflecting the light is relatively close (e.g. within 2 m).
The time to digital value convertor 114 may be capable of receiving outputs from multiple pixels simultaneously and converting these into digital values for storage in the histogram memory. In an alternative approach, multiplexing may be used such that the time to digital convertor 114 receives signals from each pixel a-p of the macropixel 235 in a series. In one example, a raster scan of the pixels a-p may be used, such that signals are received from pixel a, then pixel b, etc. Where this is done, the histogram memory may record data from all the pixels a-p together with identifiers which identify which from pixel the data was received.
Data is periodically transferred to the image processor 106. Transfer for a given pixel may occur when a peak in the data for that pixel has an acceptable signal to noise ratio (e.g. above a predetermined threshold). Alternatively, data transfer for a given pixel may occur after predetermined time intervals. Data transfer for all of the pixels 235a-p may occur at the same time, or may occur at different times for different pixels. The timing of data transfer may depend for example on the size of the histogram memory and extent to which multiplexing is being used. Data transfer should occur before the histogram memory becomes over-filled.
The logic circuit 118 monitors the data held in the histogram memory to determine whether peaks are present in the data. Peaks which are identified are compared by the logic circuit 118 with a threshold time value indicative of a predetermined object distance (e.g. 1.33×10−8 seconds). If a peak with a time value of less than 1.33×10−8 seconds is identified then the logic circuit may instruct that the macropixel continues to be operated in the second mode. On the other hand, if no peak with a time value of less than 1.33×10−8 seconds is identified then the logic circuit may instruct that the macropixel switch to operating in the first mode.
The image processor 106 generates a depth map which consists of a combination of individual pixel outputs and macropixel outputs. Time of flight sensing is performed with high resolution data and low resolution data being captured in parallel.
The embodiment of the invention is advantageous because it provides a relatively high signal to noise but low resolution output when a sensed object is relatively distant, but automatically switches to a higher resolution output when a closer object is present. The switching from low resolution to high resolution is performed on-the-fly. That is, it does not depend upon receiving instructions from the image processor 106 after a depth map (or other image) has been generated. Instead, switching is performed by the logic circuit 118 based upon data in the histogram memory 116 (and thus may be performed quickly). Similarly switching from high resolution to low resolution is also performed on-the-fly via instruction from the logic circuit 118, and does not depend upon receiving instructions from the image processor 106.
Embodiments of the invention provide lower resolution data for a background of an image and higher resolution data for a foreground, without requiring that two full images are obtained. As noted further above, obtaining two full images is disadvantageous because it requires the use of more energy in order to generate the illumination for those two images.
An alternative embodiment of the invention is depicted schematically in
Referring to
In
As with the method depicted in
The embodiment of
If a peak indicative of an object at a distance of less than 2 metres is identified at step 406, then at step 410 operation of the macropixel switches such that outputs from single pixels are acquired and stored separately in the histogram memory. This may be referred to as the second mode of operation. This high resolution acquisition step repeats multiple times so that more high resolution data is obtained. The data is transferred to the image processor 106 and the method is restarted. The low resolution data that was acquired before switching to the second mode may be transferred to the image processor 106 as well as the high resolution data.
If a peak indicative of an object at a distance of between 2 and 5 metres is identified at step 406, then outputs from sub-groups of four pixels of the macropixel are combined together and stored separately in the histogram memory. This is step 412 and may be referred to as the third mode of operation. This intermediate resolution acquisition step repeats multiple times so that more intermediate resolution data is obtained. Periodically the data is transferred to the image processor 106 and the method is restarted. The low resolution data that was acquired before switching to the second mode may be transferred to the image processor 106 as well as the intermediate resolution data.
In an embodiment, once the switch to the second mode of operation or third mode of operation has occurred, the logic circuit 118 (see
In an embodiment, acquisition of data in the first mode of operation may continue even after a peak corresponding to an object less than 5 m or less than 2 m has been identified. The acquisition of data in the first mode of operation may continue until a desired ratio of signal to noise has been achieved. At this point operation may switch to the second or third modes. An advantage of this approach is that data with a sufficiently high signal to noise ratio will always be available for use by the image processor, even if the second or third mode does not generate useable data (i.e. data with a sufficiently high signal to noise ratio).
Modifications to the above method may be made. For example, instead of always beginning with a low resolution acquisition for each macropixel, the initial acquisition for a macropixel may be at whatever resolution was previously used for the macropixel. In one example, if a video stream is being generated then previously recorded distance data for that macropixel may be used to predict the best acquisition resolution to use for the next initial acquisition for that macropixel.
Although the described embodiments of the invention use threshold distances of 2 metres and optionally 5 metres, in other embodiments other threshold distances may be used. A method according to an embodiment of the invention may use a single threshold distance, may use two threshold distances, or may use more than two threshold distances. For example three or more threshold distances may be used.
In the described embodiments, the thresholds are mostly expressed in terms of distances. However, thresholds may be applied in terms of time instead of distances. For example the thresholds may be expressed as 1.33×10−8 seconds (equivalent to 2 metres) and as 3.325×10−8 seconds.
In the described embodiments of the invention each macropixel consists of 16 individual pixels. In other embodiments each macropixel may have a different number of individual pixels. Different groupings of those pixels may be used. For example, sub-groups of pixels consisting of more than four pixels may be used.
Embodiments of the invention allow the capture in parallel of higher resolution foreground with more spatial details together with lower resolution background at acceptable signal to noise levels.
Applications of the invention include image segmentation, for example for a video conferencing call. During a video conferencing call, images may be segmented between a foreground which is represented in higher resolution and a background which is represented in lower resolution. This may for example be done if the background is not relevant to the call and the caller wishes it to be suppressed for privacy reasons (e.g. a work related call done at home). In order to achieve this in current systems, two dimensional spatial images (as opposed to depth maps) are obtained, and image processing is used to determine which part of those 2D spatial images are foreground and which are background. This approach uses a lot of processing of power, and therefore will deplete significant energy from the battery of a mobile device. As explained elsewhere, energy depletion in a mobile device is undesirable. This intensive processing is avoided by embodiments of the invention.
By providing background measurements and foreground measurements in parallel, embodiments of the invention allow more accurate perspective and details to be provided in augmented reality systems. The same may also apply for virtual reality or mixed reality systems.
Embodiments of the invention may advantageously allow objects to be tracked more accurately. The movement of an object measured in pixels per second across a sensor will be faster for a relatively close object and slower for a relatively distant object (if the objects are travelling at the same speed). Since embodiments of the invention scale resolution according to the distance to an object, the same object travelling at the same speed can be tracked at different distances with different resolutions (giving effectively the same tracking information). Embodiments of the invention allow an object to be tracked effectively, even as the object moves through different distances from the sensor array. Embodiments of the invention advantageously provide three dimensional tracking (which is more useful than conventional two dimensional tracking). Embodiments of the invention may provide tracking over a greater range of distances (which may be referred to as depths) from the sensor array because switching between different resolutions can occur automatically.
In
It is not essential that all of the elements of the electrical circuits are in the integrated circuit. One or more of the elements may be located remotely from the integrated circuit.
For example, the time to distance converter may form part of a different integrated circuit. However, providing all of the elements in the integrated circuit may be the most efficient configuration.
In general, a logic circuit 118 may be provided for each group of pixels (which may be referred to as a macropixel).
Embodiments of the present disclosure can be employed in many different applications including, for example, in mobile phones.
100—Time of flight sensor system
102—Light source
104—Sensor module
106—Image processor
110—Sensor electronics
112—Front end
114—Time to digital value convertor
116—Memory
118—Logic circuit
120—Time to distance convertor
122, 222, 322—Sensor
131-139—Groups of sensor pixels (macropixels)
141-149—Electrical circuits
231-239—Groups of sensor pixels (macropixels)
331-339—Groups of sensor pixels (macropixels)
a-p—Pixels of a macropixel
402—Start of method
404—Low resolution acquisition
406—Check of acquired data by logic circuit
408—Further low resolution acquisition
410—Acquisition from single pixels
412—Acquisition from groups of four pixels
The skilled person will understand that in the preceding description and appended claims, positional terms such as ‘above’, ‘along’, ‘side’, etc. are made with reference to conceptual illustrations, such as those shown in the appended drawings. These terms are used for ease of reference but are not intended to be of limiting nature. These terms are therefore to be understood as referring to an object when in an orientation as shown in the accompanying drawings.
Although the disclosure has been described in terms of preferred embodiments as set forth above, it should be understood that these embodiments are illustrative only and that the claims are not limited to those embodiments. Those skilled in the art will be able to make modifications and alternatives in view of the disclosure which are contemplated as falling within the scope of the appended claims. Each feature disclosed or illustrated in the present specification may be incorporated in any embodiments, whether alone or in any appropriate combination with any other feature disclosed or illustrated herein.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/SG2020/050710 | 12/2/2020 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 62944009 | Dec 2019 | US |
