Patent Grant
12228684
This application is a U.S. National Phase of International Patent Application No. PCT/JP2019/027353 filed on Jul. 10, 2019, which claims priority benefit of Japanese Patent Application No. JP 2018-148529 filed in the Japan Patent Office on Aug. 7, 2018. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.
The present disclosure relates to rangefinders, rangefinding systems, rangefinding methods, and programs. In particular, the present disclosure relates to a rangefinder that measures the distance to an object using a time-of-flight (ToF) technique, and a rangefinding system, a rangefinding method, and a program, for implementation thereof.
The time-of-flight (ToF) technique is known as the technique for measuring the distance to an object or the three-dimensional shape of the object.
The ToF technique is the technique of irradiating an object with light and analyzing the reflected light to measure the distance to the object or the shape of the object.
A rangefinder using the ToF technique employs what is called a ToF camera in which a light-emitting unit (a light source) and a light-receiving unit (camera) are integrated. The rangefinder has a configuration of irradiating an object to be measured (a rangefinding target object) with output light from the light source and receiving the reflected light with a camera.
However, in this configuration, the light output from the light source integrated with the camera follows a round-trip path in which the light reaches the object, is reflected from the object's surface, and returns to the camera side. When the distance from the camera to the object is d, the optical path is 2d that is twice the distance d. Thus, a longer optical path reduces the intensity of the light input to the camera, causing a problem with lower rangefinding accuracy. In particular, in measuring the distance to an object far from the camera, the decrease in measurement accuracy is remarkable.
Patent Document 1 (Japanese Patent Application Laid-open No. 2018-31607), which is an earlier patent application by the same applicant, discloses a device in which the output light of the light source configured independently from and located away from the camera is directly received by the camera to measure the distance from the camera to the light source.
This configuration allows, in the case where the distance between the light source and the camera is d, the output light of the light source also to enter the camera via only the path of the distance d, making it possible to prevent the decrease in measurement accuracy due to the reduction in light intensity described above.
However, the configuration disclosed in Patent Document 1 is based on the assumption that the camera directly observes the light source. Thus, it fails to be used unless the dynamic range of the sensor is wide. For example, direct observation of the light source necessitates short exposure time to avoid saturation but causing a problem of blacking out the surrounding subjects. Besides, it does not teach a solution to the problem caused by the clock deviation between the light source and the camera.
The present disclosure is made in view of, for example, the problems described above. It is directed to provide a rangefinder, rangefinding system, rangefinding method, and program, capable of preventing a decrease in measurement accuracy due to reduced light intensity even for common objects other than a light source.
Furthermore, one embodiment of the present disclosure places a light-emitting unit (light source) and a light-receiving unit (camera) at separate positions, allowing the light-emitting unit (light source) and the light-receiving unit (camera) to be controlled with individual clocks. Accordingly, it provides a rangefinder, rangefinding system, rangefinding method, and program, allowing for measurement of a distance to an object and three-dimensional shape of the object using the ToF technique by causing the reflected light of the light emitted from the light-emitting unit (light source) onto the object to be directed to the light-receiving unit (camera).
According to a first aspect of the present disclosure, there is provided a rangefinder including:
Further, according to a second aspect of the present disclosure, there is provided a rangefinding system including:
Further, according to a third aspect of the present disclosure,
Further, according to a fourth aspect of the present disclosure,
Further, according to a fifth aspect of the present disclosure,
Note that the program of the present disclosure is, for example, a program accessible as a storage medium or a communication medium provided in a non-transitory computer-readable form to an image processing apparatus or a computer system executable of various program codes. Such a program provided in the non-transitory computer-readable form makes it possible for the processing in accordance with the program to be implemented on the information processing apparatus or the computer system.
Still other objects, features, and advantages of the present invention will become apparent from a detailed description based on embodiments of the present disclosure as described later and accompanying drawings. Note that the term “system” herein refers to a logical component set of a plurality of apparatuses and is not limited to a system in which apparatuses of the respective components are provided in the same housing.
One embodiment of the present disclosure achieves a configuration of controlling a light source unit having a plurality of light sources and a rangefinder (camera) by individual clocks to allow for setting the light sources at optional positions.
Specifically, for example, the output light of a plurality of light sources of the light source unit is reflected from an object's surface, and a ToF sensor captures the reflected light to calculate the distance to the object. A sensor clock that controls the image capturing timing of the ToF sensor is the clock specific to the rangefinder, which is different from a light source unit clock that controls the light emission timing of the light sources of the light source unit. The rangefinder creates a difference calculation formula of a plurality of distance calculation formulas used to calculate the distance from each light source to the rangefinder to calculate the distance to the object using the created difference calculation formula.
The present configuration achieves a configuration of controlling a light source unit having a plurality of light sources and a rangefinder (camera) by individual clocks to allow for setting the light sources at optional positions.
Note that the effects described in the present specification are merely examples and are not limited, and there may be additional effects.
A description is now in detail given of a rangefinder, a rangefinding system, a rangefinding method, and a program of the present disclosure with reference to the drawings. Moreover, the description is given according to the items described below.
The description is now given of an overview of rangefinding processing by the ToF technique.
As described above, the time-of-flight (ToF) technique is one of the techniques for measuring the distance to an object or the object's three-dimensional shape. The ToF technique is the technique of irradiating an object with light and analyzing the reflected light to measure the distance (depth) to the object or the object's shape.
The overview of the distance (depth) measurement processing by the ToF technique is described with reference to
This configuration measures time Δt until the output light from the light source 1 is reflected from the object 3 and is incident on the camera 2, allowing a distance (depth) d from the camera 2 to the object 3 to be measured.
The distance d can be calculated according to (Formula 1) as follows:
d=(½)×c×Δt (Formula 1)
Moreover, in the figure, the light source 1 and the camera 2 are shown at slightly distant positions for clarity. However, in a common device in the related art, the light emission timing of the light source 1 and the image capturing timing by the camera 2 are controlled by one clock, so the light source 1 and the camera 2 are configured at substantially the same position such as in the same device. Thus, the time Δt until the output light from the light source 1 is reflected from the object 3 and is incident on the camera 2 is the time for the light to travel a distance twice the distance (depth) d from the camera 2 to the object 3. This is the reason why “(½)” is multiplied in the formula expressed above for calculating the distance d in (Formula 1).
However, in the configuration illustrated in
This processing is described with reference to
This configuration allows the difference in time between the pulsed light output from the light source 1 and the pulsed light input to the camera 2 to be converted into a phase difference for observation.
The light source 1 blinks at high speed at the known frequency f [Hz]. In other words, one cycle of the light emission pattern of the light source 1 is 1/f [second]. The camera 2 measures a phase of the light blinking pattern for each pixel. A specific example of this pixel-based phase measurement configuration is described in the subsequent stage with reference to
For example, it is assumed that the difference in phase between the pulsed light output by the light source 1 and the pulsed light received by the camera 2 is φ. In this case, time Δt until the output light from the light source 1 is reflected from the object 3 and is incident on the camera 2 can be calculated by (Formula 2) as follows.
Δt=(1/f)×(φ/2π) (Formula 2)
Substituting the time Δt calculated by (Formula 2) into (Formula 1) previously described makes it possible to calculate the distance d from the camera 2 (or the light source 1) to the object 3 using (Formula 3) as follows:
d=(cφ)/4πf (Formula 3)
The phase calculation configuration of the pulsed light in pixel units in the camera 2 is described with reference to
The camera 2 described with reference to
In measuring the distance to an object using the ToF camera,
The phase of 0 degrees of (c1) is set such that ON timing (light reception timing) is the phase of the pulsed light output by the light source 1, that is, the same phase as the light emission pattern of (a) shown on the left side of
The phase of 90 degrees of (c2) is set such that the ON timing (light reception timing) is 90 degrees behind the pulsed light (light emission pattern of (a)) output by the light source 1.
The phase of 180 degrees of (c3) is set such that the ON timing (light reception timing) is 180 degrees behind the pulsed light (light emission pattern of (a)) output by the light source 1.
The phase of 270 degrees of (c4) is set such that the ON timing (light reception timing) is 270 degrees behind the pulsed light (light emission pattern of (a)) output by the light source 1.
The camera 2 sequentially executes these four types of switching to acquire light-received amounts with the light reception timing changed. In other words, four types of light-received amounts with different accumulation phases and electric charges depending on the light-received amounts are obtained.
For example, the electric charges accumulated in the case where the phase differences of the light-receiving pixels for the light emission patterns are 0 degrees, 90 degrees, 180 degrees, and 270 degrees are assumed to be set to Q0, Q90, Q180, and Q270, respectively.
In this connection, the phase difference φ used to calculate the distance d to the object by applying (Formula 3) described above is the difference in phase between the pulsed light output from the light source 1 and the pulsed light received by the camera 2. This phase difference φ can be calculated on the basis of (Formula 4) as follows.
φ=Arctan((Q90−Q270)/(Q180−Q0)) (Formula 4)
The phase difference φ calculated by (Formula 4) above, that is, the phase difference φ between the pulsed light output from the light source 1 and the pulsed light input to the camera 2, is substituted into (Formula 3). Accordingly, it is possible to calculate the distance d from the camera 2 (or the light source 1) to the object 3 from (Formula 3).
Moreover, an image frame captured by the ToF camera as illustrated n
The captured images with phase settings of the phase deviation values of 0 degrees, 90 degrees, 180 degrees, and 270 degrees for the light emission pattern (pulse) are sequentially and repeatedly captured.
One set of combinations of components with phase settings of 0 degrees, 90 degrees, 180 degrees, and 270 degrees is referred to as a frame.
In other words, one frame has information regarding the accumulated charge of Q0, Q90, Q180, and Q270 of the captured image with phase settings of phase deviation values of 0 degrees, 90 degrees, 180 degrees, and 270 degrees for the light emission pattern (pulse).
Moreover, the time between components is typically set shorter than the time between frames.
An overview of the processing executed by a rangefinder of the present disclosure is now described.
As described with reference to
The accurate detection of the phase difference can be necessary to find out the ON/OFF timing of the light emission pulse of the light source and the ON/OFF timing of pixels executed in the camera.
With this view, for example, the ON/OFF timing of the light emission pulse of the light source and the ON/OFF timing of pixels executed in the camera is desirable to be controlled by one clock, i.e., synchronized with each other by one clock.
As illustrated in
However, in the arrangement illustrated in
The arrangement illustrated in
In the arrangement illustrated in
However, in the case where the light source 1 and the camera 2 are separated in positions from each other as described above and these devices are used as separate devices, the respective devices can be necessary to be controlled by an individual clock signal.
If the light source 1 and the camera 2 are controlled by different clock signals as described above, it is difficult to completely match the control timings of the light source 1 and the camera 2, that is, to synchronize them. If no synchronization is made, components of captured images with phase settings of phase deviation values of 0 degrees, 90 degrees, 180 degrees, and 270 degrees for the light emission pattern (pulse) fail to be acquired accurately.
Moreover, types of synchronization processing that can be necessary to perform accurate rangefinding include a plurality type of synchronization processing as follows.
For example, as illustrated in
Light(t)=cos(fLt+θL)
The phase synchronization processing can be necessary to estimate the light-emitting start time t1 in addition to fL and θL of the light source 1 that emits light according to the sine wave mentioned above, as illustrated in
In other words, it is a system enabling the clock used to control the light emission timing of the light source and the clock used to control the image capturing timing of the camera to be separate clocks.
Such a configuration enables the light source to be placed at any position away from the camera. This makes it possible to place the light source closer to an object to be measured for distance.
As a result, the light emitted from the light source is reflected from the object to be measured for distance. The camera that captures the reflected light is capable of capturing the reflected light for which light intensity does not decrease, enabling the achievement of highly accurate rangefinding.
As described above, the system of the present disclosure has a configuration capable of setting an optical path until the output light from the light source is reflected from an object and incident on the camera to be short.
However, in the rangefinding system of the present disclosure, the clock that determines the control timing of the light source (CLK1) and the clock that determines the control timing of the camera (CLK2) are different. Thus, processing considering the deviation between the clocks can be necessary to be performed. This processing is described in in the subsequent stage in detail.
The rangefinding system of the present disclosure illustrated in
As illustrated in
Moreover, in the embodiment illustrated in
The description is now given of the configuration and processing using the light source unit 10 having three light sources and the configuration and processing using the light source unit 10 having two light sources.
The configuration and processing using the light source unit 10 having three light sources are now described. The pulsed light output from the three light sources a to c of the light source unit 10 is reflected from the surface of the object 30 and is incident on the rangefinder (camera) 20. The rangefinder (camera) 20 has a ToF sensor similar to the ToF camera described above with reference to
The control timing of the rangefinder (camera) 20 is executed on the basis of the clock signal output by the camera-specific clock (CLK2).
An exemplary specific configuration of the light source unit 10 is now described with reference to
As illustrated in
For example, the frequency setting of the pulsed light output from the light sources a to c in the case where the reference clock is a 10 MHz clock can be set as follows:
For example, such settings described above are possible.
Moreover, the phase of each pulsed light of the light sources a to c also depends on the length of wiring from the clock 101 to the light sources a to c, so it is expected to cause some degree of deviation. Still, such deviation is assumed to be calibrated in advance. It is herein assumed for convenience that the phases of the plurality of light sources a to c are the same, and there is no deviation. Moreover, even when the phases of the respective light sources deviate, it can be calibrated and used in the subsequent stage if the degree of deviation is known.
Further, as illustrated in the arrangement example of light sources in
For example, the light sources a to c are fixed to the housing of the light source unit 10 so that the distance between the light sources may be “W”. The light source unit 10 has a configuration in which a plurality of light sources is physically fixed.
As illustrated in the arrangement example of the light sources in
The light source control unit 102 and the delay adjustment units 103 to 105 adjust the light emission timing of the light sources a to c, that is, the light emission pulse. Moreover, the light source control unit 102 and the delay adjustment units 103 to 105 control the light emission timing of the light sources a to c by the clock supplied from the light source unit clock 101. The delay adjustment units 103 to 105 control the amount of delay in the light emission timing of the light sources a to c and 105 to 108.
Moreover, the light rays from the light sources a to c of the light source unit 10 are controlled individually so that the light rays reach the rangefinder (camera) 20 without interfering with each other. Thus, for example, the driving of the light source is controlled as follows:
In the example illustrated in
Moreover, a light-out period is set so that it is possible to identify which light source is currently emitting. After the lapse of the light-out period, the light sources are caused to emit light rays in the order of the light source a, the light source b, and the light source c. Then, also after a predetermined light-out period, the light emission repeats in the order of the light sources a to c.
The configuration of the rangefinder (camera) 20 is now described in detail with reference to
As illustrated in
The ToF sensor 202 includes an image sensor similar to the ToF camera described above with reference to
The light reception timing of the ToF sensor 202 is determined by the clock signal output from the sensor clock 201.
A moving subject detection unit 203 detects a moving subject from the image captured by the ToF sensor 202. This processing for the moving subject detection is used in performing the frequency synchronization processing described above, that is, the synchronization processing for modifying the clock deviation between the light source and the ToF sensor. This processing is described in the subsequent stage in detail.
The phase calculation unit 204 calculates the phase difference φ between the light emission pulse and the light reception pulse described above with reference to
The phase image memory 205 has a frame image and phase difference data stored therein. The frame image is formed by a component (phase image) captured by the ToF sensor 202, and the phase difference data is calculated by the phase calculation unit 204.
The light source position estimation unit 206 executes processing for estimating the position and orientation of the light source unit 10, more specifically, for example, three-dimensional position estimation processing of the light sources a to c. The specific processing example is described later.
The light source unit position and orientation memory 207 stores the position and orientation information (information regarding three-dimensional positions of the light sources a to c) of the light source unit 10, estimated by the light source position estimation unit 206.
The clock phase difference (light source clock offset) calculation unit 208 performs processing for calculating the deviation between the sensor clock 201 and the light source unit clock 101, that is, the clock phase difference, and stores the calculated clock phase difference as the clock phase difference (light source clock offset) memory 209. This processing is described later in detail.
The clock deviation amount estimation unit 210 estimates an amount of deviation in frequency between the sensor clock 201 and the light source unit clock 101. The sensor clock adjustment unit 211 adjusts the frequency of the sensor clock 201 on the basis of the amount of deviation between the sensor clock 201 and the light source unit clock 101 that is estimated by the clock deviation amount estimation unit 210. In other words, the clock frequency of the sensor clock 201 and the clock frequency of the light source unit clock 101 are adjusted to eliminate or reduce the deviation between the frequencies. This processing is described later in detail.
Moreover, even if the clock frequency is adjusted, the difference in phase (phase deviation) between the two clocks is difficult to eliminate. In practical rangefinding, processing considering the clock phase difference can be necessary. The clock phase difference (light source clock offset) calculation unit 208 described above calculates this clock phase difference.
The distance (depth) calculation unit 220 calculates the distance (depth) to the object 30 to be measured. The calculated distance is the distance from the rangefinder (camera) 20 to the surface of the object 30.
For this distance calculation, the data stored in the phase image memory 205, the light source unit position and orientation memory 207, and the clock phase difference (light source clock offset) memory 209 are used. In other words, the distance calculation is performed using the data below:
This processing is described later in detail.
The processing sequence executed by the rangefinder (camera) 20 of the present disclosure is now described.
The description is initially given of an overview of the entire sequence of the rangefinding using the rangefinding system of the present disclosure having the configuration illustrated in
Moreover, the procedure illustrated in
The processing of each step of the flowchart illustrated in
(Step S11)
In step S11, the frame synchronization processing is initially executed.
The frame synchronization processing is, as described above with reference to
In step S11, the light emission timing on the side of the light source unit 10 for each of the frames and components acquired on the side of the rangefinder (camera) 20 is acquired.
Moreover, the specific processing is described in the subsequent stage.
(Step S12)
In step S12, the frequency synchronization processing is now executed. This processing is used for checking and adjusting the frequency deviation between the sensor clock used on the side of the rangefinder (camera) 20 and the light source unit clock used on the side of the light source unit 10 to eliminate the deviation in frequencies. This processing can be necessary for the rangefinder (camera) 20 to estimate the frequency of the clock used in the light source unit 10.
(Step S13)
Subsequently, in step S13, the position or orientation of the light source unit 10 is estimated. As described above with reference to
(Step S14)
Subsequently, in step S14, the distance (depth) measurement processing of the object 30 is executed. Specifically, the distance from the rangefinder (camera) 20 to the object 30 is measured.
Moreover, for this processing, the processing is executed for calculating an offset C corresponding to the clock phase difference, that is, a rangefinding error, occurring due to the difference in phase between the sensor clock and the light source clock.
The specific processing operations of these steps S11 to S14 are described in the subsequent stage.
Moreover, the procedure illustrated in
In the procedure illustrated in
In the procedure illustrated in
(Step S21)
In step S21, the processing for estimating the position and orientation of the light source unit 10 and the calculation processing for the distance to the object, that is, the distance from the rangefinder (camera) 20 to the object 30, are executed in combination together.
The processing of step S21 corresponds to the processing executed by combining the processing of step S13 and the processing of step S14 according to the procedure illustrated in
Further, in the procedure illustrated in
In the procedure illustrated in
(Step S31)
In step S31, the processing for measuring the distance (depth) to the object is executed by using the clock phase difference, that is, the offset C based on the phase difference between the sensor clock and the light source clock, specifically, the value of the rangefinding error based on the clock phase difference.
The processing executed according to the procedure illustrated in
The rangefinder (camera) 20 of the present disclosure is capable of calculating the distance (depth) to the object, for example, according to the sequence as shown in
The processing operations described above are now described.
The frame synchronization processing in step S11 described with reference to the flowchart illustrated in
A specific example of the processing is described with reference to
As described above with reference to
In this way, in a case where the respective light sources are sequentially light-up in a time-division manner, the rangefinder (camera) 20 can be necessary to identify when and which light source emits light. For this reason, as shown in the lower part of
Moreover, it is assumed that the rangefinder (camera) 20 previously checks the light-up time, light-up order, and length of the light-out period of each light source. The rangefinder (camera) 20 initially detects the first light-out period after the start of image capturing. Once detection of the light-out period is possible, then it is determined which light source is currently lit by considering the periodicity (in the order of light sources, a, b, and c).
The frame synchronization processing of step S11 of the procedure illustrated in
The frequency synchronization processing in step S12 described with reference to the flowchart illustrated in
The processing in step S12 is the frequency synchronization processing. This processing is used for checking and adjusting the frequency deviation between the sensor clock used on the side of the rangefinder (camera) 20 and the light source unit clock used on the side of the light source unit 10 to eliminate the deviation in frequencies. This processing can be necessary for the rangefinder (camera) 20 to estimate the frequency of the clock used in the light source unit 10.
A specific example of this processing is described with reference to
This processing is used to adjust the sensor clock (CLK2) on the side of the rangefinder (camera) 20 so that the sensor clock (CLK2) on the side of the rangefinder (camera) 20 illustrated in
The light source unit 10 has three light sources a to c, but all of them use the same light source unit clock (CLK1), so any one light source can be used as long as it is used for the adjustment. For example, the description is given of exemplary processing using the output light of the light source a.
As illustrated in
The received signal is a frame having the components shown in the diagrammatic part (a1) on the right side of
Multiple frames having these four components are acquired in a time series.
It is herein assumed that the rangefinder (camera) 20, the light source unit 10, and the object 30 are all in a fixed position having no motion.
In this case, the phase difference cp between the pulsed light output from the light source a of the light source unit 10 and the pulsed light input to the rangefinder (camera) 20 is a difference in phase over time of light traveling the path from the light source unit 10 via the object 10 to the rangefinder (camera) 20.
There is the case where the rangefinder (camera) 20, the light source unit 10, and the object 30 are all in a fixed position having no motion and the sensor clock (CLK2) on the side of the rangefinder (camera) 20 and the light source unit clock (CLK1) on the side of the light source unit 10 match in frequency. In this case, as shown in the diagrammatic part (a21) on the right side of
However, there can be the case where the sensor clock (CLK2) on the side of the rangefinder (camera) 20 and the light source unit clock (CLK1) on the side of the light source unit 10 do not match in frequency. In other words, there is a deviation. In this case, as illustrated in the diagrammatic part (a22) on the right side of
In the case of obtaining the observation data as illustrated in the diagrammatic part (a22) of
The detailed sequence of the sensor clock adjustment processing is as follows.
The frequencies of two clocks with deviation are denoted by fL, and fS, where fL, is the light source unit clock frequency, and fS is the sensor clock frequency.
In this case, the amount of clock deviation between two clocks for T seconds is
(fL−fS)T.
In addition, the amount of phase deviation between two clocks for T seconds is
2π(fL−fS)T.
Furthermore, the amount of phase deviation (Δφ) per unit time (one second) is
Δφ=2π(fL−fS).
The amount of phase deviation (Δφ) per unit time (one second) corresponds to the slope of the graphs illustrated in the diagrammatic parts (a21) and (a22) of
The data illustrated in the diagrammatic parts (a21) and (a22) of
It is possible to calculate the difference in frequency between two clocks (fL−fS) according to the formula described above (Δφ)=2π(fL−fs)) obtained on the basis of the amount of phase deviation (Δφ) per unit time (one second) obtained from the observation data, as expressed by the following formula.
(fL−fS)=(Δφ/2π)
The rangefinder (camera) 20 performs processing for eliminating or reducing the deviation in frequency between the sensor clock and the light source unit clock (the frequency synchronization processing) on the basis of the difference in frequency between two clocks calculated according to the formula described above. A frequency f′s of a sensor clock newly updated after adjustment is the frequency obtained by the following formula.
f′s=fS+(Δφ/2π)
As described above, the frequency synchronization processing in step S12 of the procedure of
The constituent components and processing sequence used to execute the frequency synchronization processing in the rangefinder (camera) 20 are now described with reference to
The processing of each step in the flowchart of
(Step S101)
In step S101, initially, a phase image φ (t, x) is measured on the basis of one frame acquired at time t.
This processing is executed, on the basis of one frame input from the ToF sensor 202, by the phase difference calculation unit 204 illustrated in
This processing corresponds to the processing of acquiring the phase difference φ at time t of frame 1 of the diagrammatic part (a21) or (a22) of
(Step S102)
Subsequently, in step S102, initially, a phase image φ (t+Δt, x) is measured on the basis of one frame acquired at time t+Δt.
This processing is also executed, on the basis of one frame input from the ToF sensor 202, by the phase difference calculation unit 204 illustrated in
(Step S103)
Subsequently, in step S103, a moving subject position included in the acquired frame is detected.
This processing is executed in the moving subject detection unit 203 illustrated in
As described above with reference to
The processing of step S103 is performed to remove the captured pixel of such a moving subject from the object to be processed.
(Step S104)
Subsequently, in step S104, the amount of phase deviation Δφ (t, x) per unit time (one second) is calculated using the phase image φ (t, x) and the phase image φ (t+Δt, x) acquired in steps S101 and S102, respectively, on the basis of the following formula.
Δφ(t,x)=(φ(t+Δt,x)−φ(t,x))/Δt
Moreover, as described above, x represents the pixel position of the acquired image, and the amount of phase deviation Δφ (t, x) per unit time (one second) is calculated for each pixel position.
This processing is executed in the phase difference calculation unit 204 illustrated in
(Step S105)
Subsequently, in step S105, it is determined whether or not the processing of steps S101 to S104 is completed for the specified number of times as repetitive processing operations of a plurality of frames.
In the case where the specified number of times is not reached, the processing operations of step S101 and subsequent steps are repeated. In the case where the specified number of times is reached, the processing proceeds to step S106.
(Step S106)
Subsequently, in step S106, the pixel-corresponding data of the moving subject position is removed from the amount of phase deviation Δφ (t, x) for each pixel calculated in step S104, an average value of the amounts of phase deviation Δφ (t, x) corresponding to a still subject is calculated. This average value is used as the amount of phase deviation (Δφ) per unit time (one second).
This processing is executed in the clock deviation amount estimation unit 210 illustrated in
Moreover, the reason why the data corresponding to the moving subject pixel is not included is that, when the object is moving as described above, accurate frequency synchronization processing fails to be performed even by using the reflected light from the moving object.
(Step S107)
Subsequently, in step S107, a newly adjusted sensor clock frequency f's in which the current sensor clock frequency fS substantially matches the frequency of the light source unit clock is calculated according to the following formula.
f′s=fS+(Δφ/2π)
This processing is executed in the clock deviation amount estimation unit 210 illustrated in
(Step S108)
In step S108, finally, the adjusted sensor clock frequency f′s calculated in step S107 is set to the frequency of the sensor clock.
This processing is executed in the sensor clock adjustment unit 211 illustrated in
The processing executed according to the above procedure substantially eliminates the deviation in frequency between the sensor clock (CLK2) on the side of the rangefinder (camera) 20 and the light source unit clock (CLK1) on the side of the light source unit 10.
Moreover, in the present embodiment, the clock adjustment unit 211 is provided on the side of the rangefinder (camera) 20 to eliminate or reduce the deviation in frequency between the light source unit clock (CLK1) on the side of the light source unit 10 and the sensor clock (CLK2) on the side of the rangefinder (camera) 20. For example, the clock adjustment unit can be provided on the side of the light source unit 10 to perform the adjustment for eliminating or reducing the deviation in frequency between two clocks on the side of the light source unit 10.
The position and orientation estimation processing of the light source unit in step S13 described with reference to the flowchart illustrated in
Furthermore, in this item, a description is given of a specific example of the processing in step S21 described with reference to the flowchart illustrated in
The processing of the procedure illustrated in
The processing of the procedure illustrated in
The processing of step S21 corresponds to the processing executed by combining the processing of step S13 and the processing of step S14 according to the procedure illustrated in
The specific example of the processing described above is described with reference to
However, the relative position between the light source coordinate system and the camera coordinate system is unknown,
In other words, the light source coordinate system knows each light source position,
However, the phase deviation remains.
Under these preconditions, the position and orientation estimation processing of the light source unit 10, specifically, the three-dimensional position estimation processing of each of the plurality of light sources provided in the light source unit 10 is performed.
Moreover, the light source position estimation processing is executed in the light source position estimation unit 206 of the rangefinder (camera) 20 illustrated in
The light source position estimation unit 206 initially executes the first half (procedure step (S1)) of the light source position estimation processing procedure shown in the descriptive part (1) of
In other words, in the procedure step (S1), the distance is calculated (a range between the rangefinder (camera) 20 and respective light sources a to c) using the frame acquired on the basis of the light-receiving pulse of the pulse emitted by the respective light sources a to c of the light source unit 10.
The distance between the respective light sources a to c and the rangefinder (camera) 20 is expressed by (Formula 11) below.
[Math. 1]
Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (OD)}|+|{right arrow over (DE1)}|+C
Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (OD)}|+|{right arrow over (DE2)}|+C
Obs({right arrow over (uD)},{right arrow over (E3)})=|{right arrow over (OD)}|+|{right arrow over (DE3)}|+C (Formula 11)
Note that the vector notation parameter in (Formula 11) above (parameters represented with the arrow “→” above each parameter) indicates a vector from the origin of one coordinate system, for example, the light source coordinate system. In addition, the parameters in (Formula 11) above indicate the following:
Note that the vector notation is omitted in the description above for the formulas. Moreover, the vector notation is omitted in the description herein.
As described above, in the procedure step (S1), the light source position estimation unit 206 calculates, as three distance calculation formulas from the three light sources a to c provided in the light source unit 10 to the rangefinder (camera) 20,
Subsequently, the light source position estimation unit 206 executes the second half (procedure step (S2 to S3)) of the light source position estimation processing procedure shown in the descriptive part (1) of
In other words, in the procedure step (S2), a simultaneous equation of calculation formula for the difference between three distances Obs (uD, E1), Obs (uD, E2), and Obs (uD, E3) calculated in the procedure step (S1) is created, which is the simultaneous equation shown in (Formula 12) below.
[Math. 2]
Obs({right arrow over (uD)},{right arrow over (E1)})−Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (DE1)}|−|{right arrow over (DE2)}|
Obs({right arrow over (uD)},{right arrow over (E2)})−Obs({right arrow over (uD)},{right arrow over (E3)})=|{right arrow over (DE2)}|−|{right arrow over (DE3)}|
Obs({right arrow over (uD)},{right arrow over (E3)})−Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (DE3)}|−|{right arrow over (DE1)}| (Formula 12)
Subsequently, in the procedure step (S3), the simultaneous equations mentioned above are solved to determine a plurality of subject positions LDk in the light source coordinate system and their corresponding pixel positions uk.
The light sources' positions as viewed from the camera coordinates are unknown, so this equation fails to solve without any modification in the camera coordinate system. However, the light sources' positions in the light source coordinate system are known, so it is possible to solve the equation in the light source coordinate system.
Solving the simultaneous equations shown in Formula 12 above using the light source coordinate system makes it possible to determine a plurality of subject positions LDk in the light source coordinate system and their corresponding pixel positions uk.
Moreover, in this example, the plurality of subject positions LDk includes, for example, the respective positions E1, E2, and E3 of the light sources a, b, and c of the light source unit 10 and the surface positions of the object 30.
Moreover, if the correspondence between a plurality of points in a coordinate system and pixel positions projected by the camera is known, it is typically possible to determine the relative position and orientation between the coordinate system and the camera coordinate system.
In other words, solving what is called the perspective-n-point (PnP) problem makes it possible to derive the relative relationship between the light source coordinate system and the camera coordinate system, converting the subject position in the light source coordinate system into the camera coordinate system.
This processing makes it possible to determine the subject position LDk in the camera coordinate system and its corresponding pixel position uk. Thus, in the camera coordinate system, it is possible to determine the respective positions E1, E2, and E3 of the light sources a, b, and c and their surface positions on the object 30 from the positions of captured pixels of the light sources a to c. The determination of the surface position of the object 30 in the camera coordinate system makes it possible to calculate the distance to the object 30, that is, the distance between the rangefinder (camera) 20 and the object.
The detailed configuration of the light source position estimation unit 206 of the rangefinder (camera) 20 and the processing sequence of the light source position and orientation estimation processing and the subject position calculation processing (i.e., distance calculation processing) are now described with reference to
The processing of each step in the flowchart of
(Step S201)
Step S201 initially calculates a distance (the distance between the light sources a to c and the rangefinder (camera) 20) using a frame acquired on the basis of the output light from one light source among the light sources a to c of the light source unit 10, that is, the light-receiving pulse of the emission pulse.
The distance between the respective light sources a to c and the rangefinder (camera) 20 is expressed by (Formula 11) described above.
[Math. 3]
Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (OD)}|+|{right arrow over (DE1)}|+C
Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (OD)}|+|{right arrow over (DE2)}|+C
Obs({right arrow over (uD)},{right arrow over (E3)})=|{right arrow over (OD)}|+|{right arrow over (DE3)}|+C (Formula 11)
As described above, in the procedure step S201, the light source position estimation unit 206 calculates, as three distance calculation formulas from the three light sources a to c provided in the light source unit 10 to the rangefinder (camera) 20,
The processing of step S201 is executed in a phase-to-distance conversion unit 251 illustrated in
(Step S202)
Step S202 determines whether or not the acquisition of distance information for all light sources is completed.
When the acquisition of the distance information for all the light sources is not completed, the processing returns to step S201, and the distance information acquisition processing for the light sources to be processed is executed.
If the acquisition of distance information for all the light sources is completed, the processing proceeds to step S203.
(Step S203)
In step S203, the processing for determining the position of a plurality of points of interest (subjects) in the light source coordinate system and the coordinates of the points on the image is performed.
This processing corresponds to the processing (steps S2 to S3) of
In other words, initially, a simultaneous equation of calculation formulas for the difference between the distances corresponding to the three light sources Obs (uD, E1), Obs (uD, E2), and Obs (uD, E3) calculated in step S201 is created, which is the simultaneous equation shown in (Formula 12) described above.
[Math. 4]
Obs({right arrow over (uD)},{right arrow over (E1)})−Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (DE1)}|−|{right arrow over (DE2)}|
Obs({right arrow over (uD)},{right arrow over (E2)})−Obs({right arrow over (uD)},{right arrow over (E3)})=|{right arrow over (DE2)}|−|{right arrow over (DE3)}|
Obs({right arrow over (uD)},{right arrow over (E3)})−Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (DE3)}|−|{right arrow over (DE1)}| (Formula 12)
Moreover, this difference calculation processing is executed in a difference calculation unit 252 illustrated in
The difference data calculated by the difference calculation unit 252 is output to a subject position calculation unit 253. The subject position calculation unit 253 solves the simultaneous equation to calculate a plurality of subject positions LDk in the light source coordinate system and their corresponding pixel positions uk.
As described above with reference to
Solving the simultaneous equations shown in Formula 12 above using the light source coordinate system makes it possible to determine a plurality of subject positions LDk in the light source coordinate system and their corresponding pixel positions uk.
Moreover, in this example, the plurality of subject positions LDk includes, for example, the respective positions E1, E2, and E3 of the light sources a, b, and c of the light source unit 10 and the surface positions of the object 30.
(Step S204 to S205)
As described above with reference to
First, in step S204, solving the perspective-n-point (PnP) problem makes it possible to derive the relative relationship between the light source coordinate system and the camera coordinate system.
This processing is executed in a light source-camera positional relationship estimation unit 254 illustrated in
Subsequently, in step S205, the subject position in the light source coordinate system is converted into values in the camera coordinate system. This processing is executed in the coordinate conversion unit 257.
These processing make it possible to determine the subject position LDk in the camera coordinate system and its corresponding pixel position uk. Thus, in the camera coordinate system, it is possible to determine the respective positions E1, E2, and E3 of the light sources a, b, and c and their surface positions on the object 30 from the positions of captured pixels of the light sources a to c, and to calculate the distance to the object 30, that is, the distance between the rangefinder (camera) 20 and the object.
Moreover, this distance (depth) calculation processing is executed in the distance (depth) calculation unit 220 of the rangefinder (camera) 20 described with reference to
A description is now given of a specific processing example in the case of performing the processing of step S14 described with reference to the flowchart illustrated in
The processing of the procedure illustrated in
Moreover, this processing can be executed even for the light source unit 10 provided with two light sources. In other words, the light source unit 10 is implementable as long as it includes two or more light sources.
The specific example of the processing described above is described with reference to
The phase difference calculation processing between the clock on the side of the light source unit 10 and the clock on the side of the rangefinder (camera) 20 is executed under the preconditions as follows:
Under these preconditions, the phase difference calculation processing for the clock on the side of the rangefinder (camera) 20 is executed, and further calculates the distance to a subject, such as the object 30.
Moreover, the clock phase difference calculation processing is executed in the clock phase difference (light source clock offset) calculation unit 208 of the rangefinder (camera) 20 illustrated in
Moreover, the clock phase difference (light source clock offset) calculation unit 208 calculates the phase difference with the clock on the side of the rangefinder (camera) 20 or the offset C corresponding to the rangefinding error based on the phase difference. The distance (depth) calculation unit 220 applies this offset C to calculate the distance to the object 30.
The offset C is the offset C corresponding to an error amount in calculating the distance based on the phase difference between the sensor clock 201 and the light source unit clock 101 and has the similar value to the offset C of (Formula 11) used in the procedure step (S1) described above with reference to
It is possible to execute the phase difference calculation processing with the clock on the side of the rangefinder (camera) 20 and the subject distance calculation processing of the object 30 or the like using only the light emission pulses from the two light sources of the light source unit 10.
As described above, this processing is executable not only in the configuration in which the light source unit 10 has three light sources but also in the configuration having only two light sources.
The two light sources are the light source a located at position E1 and the light source b located at position E2. Moreover, the positions are assumed to be acquired in the camera coordinate system having an origin O at the position of the rangefinder (camera) 20 illustrated in
The diagrammatic part on the right side of
At first, in the procedure step (S1), the distance is calculated (a range between the rangefinder (camera) 20 and respective light sources a and b) using the frame acquired on the basis of the light-receiving pulse of the pulse emitted by the two light sources a and b of the light source unit 10.
The distance between the respective light sources a and b and the rangefinder (camera) 20 is expressed by (Formula 21) below.
[Math. 5]
Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (OD)}|+|{right arrow over (DE1)}|+C
Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (OD)}|+|{right arrow over (DE2)}|+C (Formula 21)
Note that the vector notation parameter in (Formula 21) above (parameters represented with the arrow “→” above each parameter) indicates a vector from the origin of one coordinate system, for example, the camera coordinate system. In addition, the parameters in (Formula 21) above indicate the following:
Note that the vector notation is omitted in the description above for the formulas.
As described above, in the procedure step (S1), the distance (depth) calculation unit 220 calculates, as two distance calculation formulas from the two light sources a and b provided in the light source unit 10 to the rangefinder (camera) 20,
Subsequently, in the procedure step (S2), a calculation formula for a difference between two distances Obs (uD, E1) and Obs (uD, E2) calculated in the procedure step (S1) is created, which is the difference calculation formula expressed as Formula 22 below.
[Math. 6]
Obs({right arrow over (uD)},{right arrow over (E1)})−Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (DE1)}|−|{right arrow over (DE2)}| (Formula22)
The creation of this difference calculation formula makes it possible to eliminate the offset term C included in the distance calculation formula created in the procedure step (S1), that is, the offset term C corresponding to the rangefinding error caused by the phase deviation between clocks of the light source unit 10 and the rangefinder (camera) 20.
Subsequently, in the procedure step (S3), the distance to the object 30, that is, the distance D between the rangefinder (camera) 20 and the object 30, is calculated.
In this processing, initially, a unit vector d in the direction of a vector OD from the origin O of the rangefinder (camera) 20 to the surface D of the object 30 is determined, and the Formula 23 below is created.
[Math. 7]
{right arrow over (OD)}=D{right arrow over (d)} (Formula 23)
(Formula 23) is a linear equation formed by a line connecting the rangefinder (camera) 20 and the object 30.
In this way, in the procedure step (S3), the distance (depth) calculation unit 220 creates
Subsequently, the position (x,y,z) of the surface D of the object 30 is calculated by using (Formula 23) and the distance difference calculation formula (Formula 22) previously created in the procedure step (S2), the distance D is calculated on the basis of the position (x,y,z).
The processing of the procedure steps S1 to S3 can be summarized as follows.
In the procedure step (S1), the distance (depth) calculation unit 220 generates, as two distance calculation formulas from the at least two light sources provided in the light source unit 10 to the rangefinder (camera) 20,
Furthermore, in the procedure step (S2), one difference calculation formula for calculating the difference between the distance calculation formulas is created.
Furthermore, in the procedure step (S3),
This processing is described with reference to
The distance difference calculation formula (Formula 22) created in the procedure step (S2) corresponds to an equation indicating one side of a two-sheet hyperboloid with the respective positions E1 and E2 of the light sources a and b as focal points. It is herein referred to as a hyperboloid for the sake of simplicity.
On the other hand, (Formula 23) created in the procedure step (S3) corresponds to an equation indicating a straight line connecting the origin O of the rangefinder (camera) 20 and the surface D of the object 30.
As illustrated in the descriptive part (A) of
(Formula 23) corresponding to the linear equation are solved
as a simultaneous equation. Thus, it is possible to calculate a point satisfying the two equations, that is, an intersection point, as illustrated in the diagrammatic part (B1) of
As illustrated in the diagrammatic part (B1) of
In this way, two equations of the difference calculation formula (Formula 22) corresponding to the hyperboloid equation and
(Formula 23) corresponds to a linear equation are solved
Moreover, as illustrated in the diagrammatic part (B2) of
In general, given that the distances from points A and B to a point P are represented by |AP| and |BP|, the set of points P having constant |AP|−|BP| is a part of the hyperboloid. In other words, the above-mentioned difference calculation formula (Formula 22) is a formula indicating a part of the hyperboloid.
For example, given that the coordinates of the point A are (0,0,c), the coordinates of the point B are (0,0,−c), and |AP|−|BP| equal to 2e as illustrated in
Moreover, an illustration represented by (Formula 24) above is divided into two curved surfaces in the +z direction and the −z direction in the figure showing the hyperboloid in
The difference calculation formula (Formula 22) described above is also equivalent to this problem, so the illustration represented by (Formula 22) is one sheet of the hyperboloid (two-sheet hyperboloid).
In this way, the distance (depth) calculation unit 220 solves two equations of the difference calculation formula (Formula 22) corresponding to the hyperboloid equation and (Formula 23) corresponding to the linear equation as simultaneous equations and calculates the position (x,y,z) of the surface D of the object 30. Furthermore, the distance (depth) calculation unit 220 calculates the distance D from the origin O of the camera coordinate system on the basis of the calculated position (x,y,z).
With reference to
In the constituent components illustrated in
The processing of each step in the flowchart of
(Step S301)
Step S301 initially calculates a distance (the distance between the light sources and the rangefinder (camera) 20) using a frame acquired on the basis of the output light from any one light source among the two light source a or b of the light source unit 10, that is, the light-receiving pulse of the emission pulse.
The processing of step S301 is executed in a phase-to-distance conversion unit 271 illustrated in
(Step S302)
Step S302 determines whether or not the acquisition of distance information for two light sources is completed.
When the acquisition of the distance information for the two light sources is not completed, the processing returns to step S301, and the distance information acquisition processing for the light sources to be processed is executed.
If the acquisition of distance information for the two light sources is completed, the processing proceeds to step S303.
In the case where the acquisition of distance information of the two light sources is determined to be completed in the processing of step S302, the distance information of a formula expressed by (Formula 21) below is obtained as the distance calculation formula between the two light sources and the rangefinder (camera) 20.
[Math. 9]
Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (OD)}|+|{right arrow over (DE1)}|+C
Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (OD)}|+|{right arrow over (DE2)}|+C (Formula 21)
(Step S303)
Subsequently, in the procedure step (S303), a calculation formula for a difference between two distances Obs (uD, E1) and Obs (uD, E2) calculated corresponding to the respective light source is created, which is the difference calculation formula expressed as (Formula 22) described above.
[Math. 10]
Obs({right arrow over (uD)},{right arrow over (E1)})−Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (DE1)}|−|{right arrow over (DE2)}|(Formula 22)
Moreover, this difference calculation processing is executed in a difference calculation unit 272 illustrated in
The difference data calculated by the difference calculation unit 272 is output to the subject position calculation unit 275.
(Step S304)
Subsequently, in step S304, a hyperboloid equation that satisfies the difference calculation formula created in step S303 is created using the known conditions and parameters.
As described earlier, the difference calculation formula (Formula 22) described above is a formula indicating a part of the hyperboloid. In other words, given that the coordinates of the point A are (0,0,c), the coordinates of the point B are (0,0,−c), and |AP|−|BP| equal to 2e as illustrated in
This is because the same formula as (Formula 24) can be obtained even in the case of |AP|−|BP|=−2e. If the sign of |AP|−|BP| is known, one of them is specified.
The difference calculation formula (Formula 22) described above is also equivalent to this problem, so the illustration represented by (Formula 22) is one sheet of the hyperboloid (two-sheet hyperboloid).
In step S304, a hyperboloid equation that satisfies the difference calculation formula created in step S303 is created using the known conditions and parameters described above.
This processing is executed in the subject position calculation unit 275 illustrated in
(Step S305)
Subsequently, in step S305, the unit vector d in the direction of the vector OD from the rangefinder (camera) 20 to the surface D of the object 30 is obtained using known conditions and parameters. A linear equation including the unit vector d, i.e., the linear equation indicated in (Formula 23) described above, is created.
Moreover, given that the pixel positions in which points (x,y,z) on a certain space are projected on the camera are (u, v), the positional relationships between them are typically defined as indicated (Formula 25) below.
Moreover, f, cv, and cu are lens parameters (internal parameters) that can be obtained in advance, and are stored in the internal parameter memory 273 illustrated in
On the other hand, if the pixel positions (u, v) projected on the camera are given, it is possible to estimate the line-of-sight direction (line-of-sight vector) observed at the pixel positions can be estimated. In other words, Formula 26 below is obtained.
Solving (Formula 26) above makes it possible to determine the unit vector d corresponding to the line-of-sight vector in the direction of the vector OD.
With this processing, the unit vector d in the direction of the vector OD from the rangefinder (camera) 20 to the surface D of the object 30 is obtained. A linear equation including the unit vector d, i.e., the linear equation indicated in (Formula 23) described below, is created.
[Math. 13]
{right arrow over (OD)}=D{right arrow over (d)} (Formula 23)
Moreover, this processing is executed in a line-of-sight vector calculation unit 274 and the subject position calculation unit 275 illustrated in
[Step S306]
In step S306, finally, the simultaneous equations formed by the hyperboloid equations created in steps S303 to S304 and the linear equations created in step S305 are solved. The intersection-point position between the hyperboloid and the straight line is calculated. The distance D from the rangefinder (camera) 20 to the surface of the object 30 is calculated on the basis of the calculated intersection-point position.
This processing calculates the intersection-point position between the hyperboloid and the straight line, which is described above with reference to
[Math. 14]
Obs({right arrow over (uD)},{right arrow over (E1)})−Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (DE1)}|−|{right arrow over (DE2)}|(Formula 22)
{right arrow over (OD)}=D{right arrow over (d)} (Formula 23)
By solving as the simultaneous equations, the position (x,y,z) of the surface D of the object 30 is calculated. This position (x,y,z) is a coordinate indicating the position of the camera coordinate system. It is possible to calculate the distance D from the origin O of the camera coordinate system on the basis of the position (x,y,z).
The processing of step S306 is executed in the subject position calculation unit 275 illustrated in
In this way, it is possible to calculate the distance to the object 30.
Moreover, although not shown in the procedure, in executing the distance calculation processing, the clock phase difference (light source clock offset) calculation unit 208 illustrated in
[Math. 15]
Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (OD)}|+|{right arrow over (DE1)}|+C
Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (OD)}|+|{right arrow over (DE2)}|+C (Formula 21)
The value of the offset C is a fixed value, so it can be used in the subsequent distance calculation processing.
Moreover, the position and distance of the object 30 are calculated by the processing according to the procedure illustrated in
Moreover, if the value of the offset C can be calculated, the difference in phase between the sensor clock and the light source clock can also be calculated on the basis of the value of the offset C. The calculated phase difference is stored in the clock phase difference (light source clock offset) memory 209, together with the value of the offset C.
Next, The description is now given of specific example of object distance (depth) calculation processing in a case of known light source position and light source clock offset.
This processing corresponds to the processing of step S31 of the flowchart illustrated in
The processing illustrated in
In other words, for example, this processing is the distance calculation processing performed after the clock phase difference (light source clock offset) calculation unit 208 stores the value of the offset C in the clock phase difference (light source clock offset) memory 209, in the processing described with reference to
Moreover, as described above, the offset C corresponds to the value of the rangefinding error that occurs due to the difference in phase between the sensor clock and the light source clock.
The specific example of the processing described above is described with reference to
In this processing, the distance (depth) calculation unit 220 calculates the distance to the object 30 by solving the simultaneous equations of the two formulas shown on the right side of
In other words, it is the simultaneous equation indicated in (Formula 31) below.
[Math. 16]
Obs({right arrow over (uD)},{right arrow over (E1)})=|{right arrow over (OD)}|+|{right arrow over (DE1)}|+C
{right arrow over (OD)}=D{right arrow over (d)} (Formula 31)
The simultaneous equations of Formula 31 above are formed by two formulas of the distance calculation formula between the light source a and the rangefinder (camera) 20 and the linear equation including the unit vector d in the direction of the vector OD from the rangefinder (camera) 20 to the surface of the object 30.
The distance calculation formula between the light source a and the rangefinder (camera) 20 is the same as the formula included in (Formula 11) and (Formula 21) described above.
Obs (uD, E1) is the distance observed at pixel position up in emitting light by the light source a located at position E1.
uD=(uD, vD) is the pixel position where point D is projected.
In addition, C is an offset term and corresponds to the value of the rangefinding error that occurs due to the difference in phase between the sensor clock and the light source clock. In this processing example, the value of this offset C is known.
The linear equation including the unit vector d is the same equation as (Formula 23) described above.
A phase-to-distance conversion unit 282 initially calculates a distance (the distance between the light source and the rangefinder (camera) 20) using a frame acquired on the basis of the output light from one light source from the light source a of the light source unit 10, that is, the light-receiving pulse of the emission pulse.
This calculation formula is the distance calculation formula for calculating Obs (uD, E1) of (Formula 31) above. Moreover, the offset C included in this distance calculation formula is the value of the rangefinding error that occurs due to the difference in phase between the sensor clock and the light source clock and is the known value stored in the clock phase difference (light source clock offset) memory 209 illustrated in
On the other hand, a line-of-sight vector calculation unit 281 and a subject position calculation unit 283 obtains the unit vector d in the direction of the vector OD from the rangefinder (camera) 20 to the surface D of the object 30 using known conditions and parameters. A linear equation including the unit vector d, i.e., the linear equation included in (Formula 31) above, is created.
This processing is the same as the processing of step S305 of the procedure described above with reference to
Subsequently, the subject position calculation unit 283 solves a simultaneous equation formed by a distance calculation formula created by the phase-to-distance conversion unit 282 and a linear equation including the unit vector d, that is, the simultaneous equation of (Formula 31) described above, and obtains the three-dimensional position of the surface D of the object 30 to calculate the distance D from the origin O of the camera coordinate system. Moreover, the light source position information used in these processing operations is acquired from the light source unit position and orientation memory 207. The value of the offset C is acquired from the clock phase difference (light source clock offset) memory 209.
Next, the description is now given of configuration of preventing occurrence of multiple solutions in solutions of simultaneous equation of hyperboloid equation and linear equation.
As described earlier with reference to
[Math. 17]
Obs({right arrow over (uD)},{right arrow over (E1)})−Obs({right arrow over (uD)},{right arrow over (E2)})=|{right arrow over (DE1)}|−|{right arrow over (DE2)}|(Formula 22)
{right arrow over (OD)}=D{right arrow over (d)} (Formula 23)
In other words, (Formula 22) and (Formula 23) form the simultaneous equations. (Formula 22) is the difference calculation formula of the distance calculation formula from the two light sources. (Formula 23) is the linear equation determined by a line connecting the rangefinder (camera) 20 and the object 30, that is, the linear equation including the unit vector d in the direction of the vector OD from the origin O of the rangefinder (camera) 20 to the surface D of the object 30.
As described with reference to the descriptive part (A) of
(Formula 23) corresponding to the linear equation are solved as a simultaneous equation.
Thus, it is possible to calculate a point satisfying the two equations, that is, an intersection point, as illustrated in the diagrammatic part (B1) of
As illustrated in the diagrammatic part (B1) of
Thus, two equations of the difference calculation equation (Formula 22) corresponding to the hyperboloid equation and (Formula 23) corresponds to a linear equation are solved as simultaneous equations. Thus, it is possible to calculate the position (x,y,z) of the surface D of the object 30. This position (x,y,z) is a coordinate indicating the position of the camera coordinate system. It is possible to calculate the distance D from the origin O of the camera coordinate system on the basis of the position (x,y,z).
However, as illustrated in the diagrammatic part (B2) of
A description is now given of the arrangement restriction of a subject used to prevent the occurrence of such multiple solutions, that is, an object to be measured for distance.
As illustrated in
Subsequently, the hyperboloid 302 tangent to the endpoint of the object (subject) existence allowable region 301 is determined. In this event, the surface where the hyperboloid 302 is tangent to the endpoint of the object (subject) existence allowable region 301 and the absolute value of k defined by the following formula (Formula 41) is the maximum value is calculated.
[Math. 18]
|{right arrow over (DE1)}|−|{right arrow over (DE2)}|=k (Formula 41)
The value k calculated by (Formula 41) above corresponds to the difference between the distance between the object 30 and the light source a and the distance between the object 30 and the light source b.
The camera, that is, the rangefinder (camera) 20, is arranged at a position as shown in the figure in the inner region of the hyperboloid defined by this condition.
Such arrangement makes it possible to create a setting in which multiple solutions are prevented from occurring in the simultaneous equations formed by the difference calculation formula (Formula 22) corresponding to the hyperboloid equation and (Formula 23) corresponding to the linear equation, that is, two intersection points are prevented from being detected.
The example illustrated in
(Condition A) Arrange light sources a and b and a camera, that is, the rangefinder (camera) 20 in a straight line, and
(Condition B) Tilt the optical axis (shooting direction) of the rangefinder (camera) 20 from the straight line described above.
The arrangement of the light sources and the rangefinder (camera) 20 satisfying these conditions A and B makes it possible to create a setting in which multiple solutions are prevented from occurring in the simultaneous equations formed by the difference calculation formula (Formula 22) corresponding to the hyperboloid equation and (Formula 23) corresponding to the linear equation, that is, two intersection points are prevented from being detected.
For example, it is possible to arrange the object 30 to be measured for distance in an object (subject) existence allowable region 303 illustrated in
An exemplary configuration using a plurality of light source units is now described. The rangefinding system of the present disclosure has technical features that the light source unit can be configured independently of the rangefinder (camera).
In other words, as illustrated in
Thus, it is possible to place the light source unit 10 near the object 30 to be measured for distance, shortening the optical path until the light from each light source of the light source unit 10 is reflected from the surface of the object 30 and is incident on the sensor (ToF sensor) of the rangefinder (camera) 20. This configuration makes it possible for the sensor (ToF sensor) of the rangefinder (camera) 20 to receive high-intensity light, improving the detection accuracy.
Moreover, the light rays from the light sources a to c of the light source unit 10 are controlled individually so that the light rays reach the rangefinder (camera) 20 without interfering with each other. Thus, for example, the driving of the light source is controlled as follows:
The use of such characteristics resulting in the configuration illustrated in
In other words, as illustrated in
The light source units U1 to U3 are connected with wirings for synchronization, and the light emission of them is controlled at a timing specified under the control of one clock.
However, the light sources a to i of each of the light source units are controlled individually so that the light rays reach the rangefinder (camera) 20 without interfering with each other. Specifically, for example, the driving of the light source is controlled as follows:
Such a configuration makes it possible to accurately measure the distances of various objects placed in a colored space. A specific usage example is illustrated in
Different articles are placed on each shelf, and the light from the light source is reflected from the surface of each article and incident on the ToF sensor of the rangefinder (camera) 20. The ToF sensor calculates the distance on the basis of the highest intensity light in the input light. Such a configuration makes it possible to measure the distance to each object at various positions in a wide space with high accuracy.
Next, the description is given of other usage examples of rangefinding system of the present disclosure.
This rangefinder is, for example, a device that is capable of detecting the posture or the like of the driver.
For example, it can be used for the emergency stop of the vehicle in the case where the driver is in a position that leans against the steering wheel due to a sudden illness.
The output light of the light source unit 10 is applied to the driver's head or the like, and its reflected light is captured by the rangefinder (camera) 20 to detect the driver's posture. When a posture different from the normal posture of the driver is detected, it is possible to take measures such as outputting an alarm or making an emergency stop of the vehicle.
The HMD displays a real-world view captured by the camera and a virtual image such as, for example, animation, by combining them.
An image such as animation is necessary to be displayed and moved by combining it with various objects existing in the actual captured image.
For example, the animation animal moves on the floor of the living room, the side of the TV, or the like shown in the figure.
The display of such a VR image on the HMD is necessary to accurately measure the distance to an actual object.
The system of the present disclosure is used for this rangefinding. For example, a ceiling light fixture shown in the figure is used as the light source unit 10 of the system of the present disclosure, and the rangefinder (camera) 20 of the present disclosure is built in the HMD worn by the user.
Such a system enables the rangefinder (camera) 20 on the side of the HMD to analyze the reflected light of the output light from the light source unit 10 configured as the ceiling light fixture and to accurately measure the distance to various objects such as the floor, television, and foliage plants in the living room. The use of the distance information for virtual object display control in the HMD makes it possible to output a highly accurate VR image.
Moreover, in the example illustrated in
Such a configuration enables the light from the light source incorporated into the HMD on the user's side to be reflected from various objects in the living room and the reflected light to be incident on the ToF sensor in the rangefinder (camera) 20 on the television. Thus, it is possible to analyze the light-receiving signal to measure the distance of each object. The rangefinding information is transmitted to the HMD or the television via, for example, Wi-Fi or the like, and a VR image using the rangefinding information is output to the HMD or the television.
The exemplary arrangement of the light source unit 10 and the rangefinder (camera) 20 can have various configurations. For example, as illustrated in
As another usage example of the rangefinding system of the present disclosure, for example, a usage example of a car wash machine is illustrated in
The car wash machine can be necessary to measure the position of the car to be washed accurately to perform the optimum washing processing.
For that purpose, the rangefinding system of the present disclosure can be used. The light source unit 10 and the rangefinder (camera) 20 are arranged on the ceiling and side surfaces of the car wash machine.
This configuration causes the light from the light source unit 10 to be reflected from the vehicle's surface and incident on the ToF sensor of the light source unit 10. The analysis of the reflected light makes it possible to accurately measure the distance to the vehicle surface, allowing for optimal control of water pressure adjustment, cleaning agent injection amount, or the like, thereby achieving more efficient cleaning.
A description is now given of an exemplary hardware configuration of the rangefinder will be described with reference to
Moreover,
A central processing unit (CPU) 501 functions as a data processing unit that executes various processes in accordance with programs stored in a read-only memory (ROM) 502 or a storage unit 508. For example, the processing according to the sequence described in the above-described embodiments is performed. The program to be executed by the CPU 501, data, and the like are stored in a random access memory (RAM) 503. The CPU 501, the ROM 502, and the RAM 503 are mutually connected via a bus 504.
The CPU 501 is connected to an input/output interface 505 via the bus 504. An input unit 506 including any of various switches, a keyboard, a mouse, a microphone, or the like, an output unit 507 including a display, a speaker, or the like, are connected to the input/output interface 505.
The CPU 501 executes various processes in response to an instruction input from the input unit 506, and outputs processing results to, for example, the output unit 507.
The storage unit 508 connected to the input/output interface 505 is configured using, for example, a hard disk and the like, and stores a program to be executed by the CPU 501 and various types of data. The communication unit 509 functions as a data communication transceiver unit via a network such as the Internet, a local area network, and furthermore as a transceiver unit for a broadcast wave, and communicates with an external device.
A drive 510 connected to the input/output interface 505 drives removable media 511 such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory such as a memory card, and executes data recording or reading.
A ToF sensor 521 detects the object reflection light of the light output from the light source. A sensor clock 522 is used as a clock to control the light reception timing of the ToF sensor 521.
The embodiments of the present disclosure have been described in detail with reference to the specific embodiments. However, it is self-evident that those skilled in the art can make modifications and substitutions of the embodiments within a scope not departing from a gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted restrictively. In order to determine the gist of the present disclosure, the scope of claims should be taken into consideration.
Moreover, the technology disclosed in the specification herein may include the following configuration.
(1) A rangefinder including:
(2) The rangefinder according to (1), further including:
(3) The rangefinder according to (2), further including:
(4) The rangefinder according to any one of (1) to (3), further including:
(5) The rangefinder according to (4), in which the light source position estimation unit
(6) The rangefinder according to (5), further including:
(7) The rangefinder according to (6), in which the difference calculation formula is
(8) The rangefinder according to any one of (1) to (7), further including:
(9) The rangefinder according to (8), in which the difference calculation formula is
(10) A rangefinding system including:
(11) The rangefinding system according to (10), in which the light source unit includes the plurality of light sources having a fixed relative position.
(12) The rangefinding system according to (10) or (11), in which the light source unit includes three or more light sources having a fixed relative position.
(13) The rangefinding system according to any one of (10) to (12), in which the rangefinding system includes
(14) The rangefinding system according to any one of (10) to (13), in which
(15) The rangefinding system according to any one of (10) to (14), further including:
(16) A rangefinding method executed in a rangefinder, the method including:
(17) A rangefinding method executed in a rangefinding system including a light source unit having a plurality of light sources and a rangefinder configured to receive light being output from each light source of the light source unit and being reflected from a surface of an object, the method including:
(18) A program causing a rangefinder to execute rangefinding processing including:
Further, the series of processing described in the specification can be executed by hardware, software, or a complex configuration of the both. In a case where the processing is executed using software, it is possible to execute the processing by installing a program recording a processing sequence on a memory in a computer built into dedicated hardware or by installing a program in a general-purpose computer that can execute various processes. For example, the program can be recorded in a recording medium in advance. In addition to installing on a computer from the recording medium, it is possible to receive a program via a network, such as a local area network (LAN) and the Internet, and install the received program on a recording medium such as a built-in hard disk.
Note that various processes described in the specification not only are executed in a time-series manner according to the description but also may be executed in parallel or separately depending on the processing performance of an apparatus that executes the process or need. Furthermore, the term “system” in the present specification refers to a logical set configuration of a plurality of apparatuses, and is not limited to a system in which apparatuses of the respective configurations are provided in the same housing.
As described above, one embodiment of the present disclosure achieves a configuration of controlling a light source unit having a plurality of light sources and a rangefinder (camera) by individual clocks to allow for setting the light sources at optional positions.
Specifically, for example, the output light of a plurality of light sources of the light source unit is reflected from an object's surface, and a ToF sensor captures the reflected light to calculate the distance to the object. A sensor clock that controls the image capturing timing of the ToF sensor is the clock specific to the rangefinder, which is different from a light source unit clock that controls the light emission timing of the light sources of the light source unit. The rangefinder creates a difference calculation formula of a plurality of distance calculation formulas used to calculate the distance from each light source to the rangefinder to calculate the distance to the object using the created difference calculation formula.
The present configuration achieves a configuration of controlling a light source unit having a plurality of light sources and a rangefinder (camera) by individual clocks to allow for setting the light sources at optional positions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2018-148529 | Aug 2018 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/027353 | 7/10/2019 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2020/031603 | 2/13/2020 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6373428 | McEwan | Apr 2002 | B1 |
| 9739609 | Lewis | Aug 2017 | B1 |
| 20100013711 | Bartlett | Jan 2010 | A1 |
| 20110176146 | Alvarez Diez | Jul 2011 | A1 |
| 20130242058 | Bae | Sep 2013 | A1 |
| 20160259032 | Hehn | Sep 2016 | A1 |
| 20160259038 | Retterath | Sep 2016 | A1 |
| 20180088218 | Kawasaki | Mar 2018 | A1 |
| 20180180733 | Smits | Jun 2018 | A1 |
| 20190011544 | Gassend | Jan 2019 | A1 |
| 20190101614 | Lohbihler | Apr 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 101981468 | Feb 2011 | CN |
| 105393083 | Mar 2016 | CN |
| 2010-190675 | Sep 2010 | JP |
| 2018-031607 | Mar 2018 | JP |
| 10-2013-0106094 | Sep 2013 | KR |
| 200905171 | Feb 2009 | TW |
| 2018125850 | Jul 2018 | WO |
| Entry |
|---|
| International Search Report and Written Opinion of PCT Application No. PCT/JP2019/027353, issued on Sep. 17, 2019, 13 pages of ISRWO. |
| Number | Date | Country | |
|---|---|---|---|
| 20210293939 A1 | Sep 2021 | US |
