Patent Application
20220366583
Depth sensing systems, such as time-of-flight (ToF) cameras, may be used to produce a depth image of an environment, with each pixel of the depth image representing a distance to a corresponding point in the environment. In ToF imaging, a distance to a point on an imaged surface in the environment is determined based on a length of a time interval in which amplitude-modulated light emitted by the depth imaging system travels out to that point and then returns back to a sensor of the depth imaging system.
Examples are disclosed that relate to determining phase orders for phase data in a depth imaging system for use in unwrapping the phase data to determine a depth value for a pixel. One example provides a computing system comprising a depth sensor comprising a plurality of pixels, each pixel configured to sense light, an illumination source configured to output light, a logic machine, and a storage machine holding instructions executable by the logic machine to control the illumination source to output amplitude-modulated light at K modulation frequencies, where K is an integer greater than or equal to two. The instructions are further executable to, for each pixel of the plurality of pixels, make K phase measurements each at a different modulation frequency of the K modulation frequencies to form a set of noisy phase measurements, determine a location at which a projection line that passes through the set of noisy phase measurements in a K-dimensional phase space passes through a lower dimensional plane in the K-dimensional phase space, the projection line being parallel to a noise free phase evolution line, compare the location to a plurality of independent terms of a predetermined matrix of points in the lower dimensional plane, each point representing a corresponding predetermined noisy phase order set with known noise, and select a set of independent terms. The instructions are further executable to locate a corresponding set of noiseless phase orders by using the set of independent terms to reference a look-up table, determine a distance value for the pixel based upon the corresponding set of noiseless phase orders, and output the distance value for the pixel.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
‘Phase-based’ optical ToF imaging is a variant of ToF imaging in which depth is computed based on the phase shift of amplitude modulated light reflected back from a subject. The phase shift is proportional to the subject's distance modulo the wavelength of the modulation frequency.
However, due to the periodic nature of the modulated light, the measured total phase repeats (or wraps) every 2π. For example, given a measured phase {tilde over (ϕ)}k, the total phase is {tilde over (ϕ)}k 2πnk, where nk is an integer. Since nk cannot be directly measured via a phase based ToF pixel, the total phase, and thus the actual distance related to the measurement, is ambiguous. As the measured phase ranges from 0 to 2π, the range of unambiguity in the determined distance is from 0 to
To address this issue, two or more different modulation frequencies k can be used to increase the range of unambiguity, allowing the phase information to be “unwrapped” for the accurate determination of distance. Phase unwrapping, or ‘de-aliasing,’ is a way to disambiguate the phase shift data and identify a distance value by illuminating the scene with amplitude-modulated light of a plurality of different frequencies, as the distance ambiguities are different for each frequency of illumination light. This extends the unambiguity range to a distance corresponding to a common beat frequency for all frequencies used.
However, due to noise in the measured phases, the phase unwrapping computation may not be straightforward, as the phase number nk (the number of the phase wrapping) for a measurement made at a modulation frequency k may be uncertain. At wrapping points, noise may cause a phase number to appear to have wrapped into a next phase order, which can result in an inaccurate determination of distance.
Accordingly, examples are disclosed relating to performing phase unwrapping to determine distance values for depth image data by generating a matrix of points and a look-up table for use in ToF phase unwrapping, and utilizing the matrix of points and look-up table to determine a phase order set and associated depth value for a noisy phase point. Briefly, a K-dimensional phase point representing K noisy phase measurements is projected onto a lower dimensional plan and compared to a predetermined matrix of points to identify independent terms. A most likely phase order set is then selected by using the independent terms to reference a look-up table. Then, a distance value is determined and output based on the most likely phase order set. The matrix of points is generated by projecting a plurality of total phase points—representing computationally-derived noisy phase orders—onto a lower dimensional plane, discussed in more detail below. An associated look-up table is further generated by plotting independent terms through the matrix of points. The examples disclosed herein may achieve improved efficiency with fast computation and low memory usage compared to graphical phase unwrapping methods that use optimization to determine a most likely phase order set.
Prior to discussing these examples in detail,
Depth imaging system 200 also includes a modulated light emitter 230, and an analog and/or digitally modulated electronic shutter 232 for sensor array 204 to control the integration of light by the sensor array 204. Modulated light emitter 230 and sensor array 204 may be controlled via controller 208. Modulated light emitter 230 may be configured to emit electromagnetic radiation having any frequency detectable by ToF pixels 206. For example, modulated light emitter 230 may include an infrared (IR) light-emitting diode (LED), laser diode (LD), or any other suitable light source. The amplitude modulated light may be modulated at different frequencies sequentially or simultaneously, e.g., the modulation waveform may comprise a manifold of frequencies.
Sensor array 204 is configured to image light from modulated light emitter 230 as reflected off surface 220 and back to the camera. Each ToF pixel 206 of sensor array 204 may comprise one or more pixel taps operable to integrate the reflected light signal at different time intervals, from which the phase shift can be determined. Sensor array 204 is controlled, for each modulation frequency, to sample light at plural phase angles of the amplitude-modulated light from the light source, and determine a phase sample for each modulation frequency from the plurality of light samples for the modulation frequency. The phase samples can then be unwrapped via the examples described herein to obtain a depth value for each pixel.
{tilde over (S)}
k(m,n)=Ãk(m,n)ei{tilde over (ϕ)}
where {tilde over (ϕ)}k is the phase, {m∈1,2 . . . ,M}, {n∈1,2 . . . ,N}, and {k∈1,2 . . . ,K}. A tilde accent over a variable indicates that the variable has been obtained and/or calculated experimentally, while the absence of a tilde accent indicates variables that correspond to a noise-free situation.
As the disclosed examples operate on a pixel-by-pixel basis, the (m,n) notation will be omitted going forward for simplicity. The terms “distance” and “depth value” may be used interchangeably when referring to the output value of a pixel. While various examples are described in the context of three light modulation frequencies, any number of frequencies K≥2 can be used.
The phase of the complex signal {tilde over (ϕ)}k may be computed as
{tilde over (ϕ)}k=arctan 2({tilde over (S)}ki,{tilde over (S)}kr) Eq. 2
where {tilde over (S)}ki is the imaginary part of the signal collected for frequency k and {tilde over (S)}kr is the real part of the signal collected. The measured phase is used to compute the depth value associated with the pixel. However, as mentioned above, in phase-based ToF imaging, there is a limitation on the distance that can be measured (referred to as the unambiguity range) imposed by the modulation frequency. At the sensor, the relationship between the total phase ϕ of the amplitude modulated light and the travelled distance is proportional to the modulation frequency of the light, as given by Equation 3:
where d is the distance, f is the modulation frequency, and c is the speed of light. As described above, due to the periodic nature of the modulated light, the measured total phase repeats (or wraps) every 2π. For example, given a measured phase {tilde over (ϕ)}k, the total phase is {tilde over (ϕ)}k+2πnk, where nk is an integer. Since nk cannot be directly measured via a phase based ToF pixel, the total phase is ambiguous. As the measured phase ranges from 0 to 2π, the range of unambiguity in the determined distance is from 0 to
Distances less than
can be determined without ambiguity, whereas distances beyond this range cannot be determined without additional information. As such, higher frequencies will have a shorter range of unambiguity.
is the distance to the first point. The phase order of the total phase is the number of times the phase wraps around. For example, line 402 corresponds to the phase in the zeroth phase order of frequency f1. Likewise, lines 404 and 406 correspond to the first and second phase orders of frequency f1.
The modulation frequency of the illumination light imposes a linear variation in precision of a phase measurement. Linear variations on the estimation of the phase lead to variations in the depth value depending on frequency. Rearranging Eq. 3 above to solve for distance results in:
where {tilde over (ϕ)}k is the measured noisy phase of modulation frequency fk and nk is the phase order. Here,
Accordingly, a set of K≥2 modulation frequencies k can be used to increase the range of unambiguity, allowing the phase information to be unwrapped for the accurate determination of distance. Phase unwrapping, which can also be referred to as ‘de-aliasing,’ is a way to disambiguate the phase shift data and identify a correct distance value by illuminating the scene with amplitude-modulated light of a plurality of different frequencies, as the distance ambiguities are different for each frequency of illumination light. For example, in a multifrequency method, the amplitude modulated light may comprise a waveform comprising a plurality of frequencies {right arrow over (f)}={f1 f2, . . . , fK}. The collection of frequencies comprises frequencies that are chosen to wrap at different locations in the unambiguity range (see
In the absence of noise, the determined distance ideally would same for each frequency fk (see Eq. 4). For example, in the example scenario in
R≡{right arrow over (ϕ)}
0
+t{circumflex over (f)} Eq. 5
where t is a parameter corresponding to the distance, {circumflex over (f)} is the slope, {right arrow over (ϕ)}0 is the initial phase, and {right arrow over (r)}0 is the starting point. The line of Eq. 5 is a phase evolution line corresponding to the ideal phase in the absence of noise. Equation 3 suggests the phase begins at 0. With this assumption, {right arrow over (ϕ)}0={right arrow over (0)} and the equation is simplified to:
R≡t{circumflex over (f)} Eq. 6
where R is the line and t is the distance parameter.
In an ideal error-free scenario, the depicted combination of three measured phases will correspond to a unique depth value within the unambiguity range. However, due to noise, the phase number nk (the number of the phase wrapping) for a measurement made at a modulation frequency k may be uncertain. As described in more detail below, an unwrapping process to obtain the most likely phase order set ({right arrow over (n)}={n1, n2, . . . , nK}) can be based on comparing a projected noisy phase point to a predetermined matrix of points in a lower dimensional projection plane. The noisy phase point in K-dimensional phase space is projected along a direction parallel to a reference phase vector, which is a line that represents the evolution of phase with distance. In one example, a noisy phase point is projected to a plane normal to the reference phase vector. In another example, the noisy phase point is projected onto a zero plane (e.g. a cartesian plane with one coordinate set at a zero value) in the K-dimensional space. The most likely phase order set is determined by comparing the projected noisy phase point to independent terms plotted through the matrix of points, and selecting the most likely phase order set by using the independent terms to reference a look-up table.
Method 500 comprises, at 502, generating total phase order sets representative of phase order sets likely to be produced in the presence of noise. Each phase order set comprises K total phases corresponding to K frequencies of amplitude modulated light. In some examples, at 504, a stochastic process is used to generate the points from the reference phase vector. In other examples, an analytical method or error propagation method may be used. In examples where the probability of noise is known, the total phase order sets may be calculated based on the probabilities. In some examples, at 506, the reference phase vector is fine sampled by generating noisy phase points from points along the reference phase vector, as described in more detail below. Method 500 further comprises, at 508, generating a set of parallel lines in K-dimensional space that run through points associated with the total phase order sets generated at 502.
At 510, method 500 comprises locating the intersection of the parallel lines with a lower dimensional plane to form a matrix of points. In some examples, at 512, the lower dimensional plane is normal to the reference phase vector. In other examples, at 514, the lower dimensional plane is a zero plane. Where the lower dimensional plane is normal to the reference phase vectors, at 516, a rotation can be performed on the matrix of points to reduce dimensionality. Additionally or alternatively, a change of coordinate system may be applied to reduce dimensionality.
Method 500 further comprises, at 518, plotting lines through the matrix of points to group points. A group of points comprises one or more points lying on or near a line of a set of parallel lines. As discussed in more detail below, each set of parallel lines in the lower dimensional plane corresponds to a different independent term. Each independent term references a line—and corresponding group of points—within the set of parallel lines. Each point lies at an intersection of lines and each point is thus referenced by the independent terms. As each point is associated with a phase order set, at 520, method 500 comprises generating a table that references phase order sets by independent terms, and at 522, storing the matrix of points and look-up table, which can then be utilized for unwrapping noisy phase data at 524.
As discussed above, a stochastic process may be used to calculate phase order sets that are most likely to be produced by the data {right arrow over (N)}={{right arrow over (n)}1, {right arrow over (n)}2, . . . {right arrow over (n)}L}. In one example employing a stochastic method, a range of interest is fine sampled as described below, and the phase vectors ({right arrow over (ϕ)}(d)) are provided via Eq. 3. The range of interest can comprise any subset up to and including the full range of unambiguity. Once the reference phase vector {right arrow over (ϕ)}(d) are generated, random noise, compatible with noise generated by the image sensor, is added to generate a noisy phase vector {right arrow over ({tilde over (ϕ)})}(d). The noise generated may comprise anisotropic noise (variance dependent of frequency) or isotropic noise (variance independent of frequency). The phase order vectors {right arrow over (n)}l are calculated as a floor:
where the floor function outputs the greatest integer less than or equal to a value. The procedure may be repeated a suitable number of times to check possible combinations of phase orders. Unique phase orders may be selected for the phase order sets N.
As mentioned above, the series of total phase triplets in
Next, a number of parallel lines are generated using, for each line, a set of noisy phase orders as the points and the phase evolution line as slope. The set of L parallel lines {right arrow over (R)}L is described as:
{right arrow over (R)}
L
≡{right arrow over (r)}
L
={right arrow over (n)}
L
+t
L
{circumflex over (f)} Eq. 8
where {right arrow over (r)}L is a line corresponding to point {right arrow over (n)}L, tL is a distance parameter, and f is the frequency vector.
The total phase points are projected onto a lower dimensional plane (e.g., a hyperplane) by extending the parallel lines to intersect the lower dimensional plane thereby obtaining a matrix of points. Each point in the matrix of points corresponds to an intersection of a line with the lower dimensional plane. Further, each point of the matrix represents a phase order set (i.e., the phase order set associated with the line). Any suitable lower dimensional plane may be used for projecting points. In some examples, a plane normal to the phase vector is used (e.g.,
In some examples, the normal plane P may be chosen to go through the origin, which may simplify calculations. With this choice, normal plane P is defined as:
P≡{circumflex over (f)}{right arrow over (r)}=0 Eq. 9
with {right arrow over (r)}L={r1,r2,r3} and where {circumflex over (f)} is the phase vector. The distance to propagate a line {right arrow over (R)}l with l∈{1,2, . . . ,L} to the plane P is calculated as the intersection of a line {right arrow over (R)}l and the plane P. In 3-dimensional space this produces a determined system with 3 equations and 3 variables. The solution can be solved in vector form:
t
l
=−{right arrow over (n)}
l
·{circumflex over (f)}
t Eq. 10
where tl is the distance in phase space between phase point {right arrow over (n)}l and the projection plane along line {right arrow over (R)}l, and {circumflex over (f)}t is the transpose of the reference phase vector. The projected points onto the plane P are calculated as:
{right arrow over (r)}
P
l
={right arrow over (n)}
l
+t
l
{circumflex over (f)} Eq. 11
where {right arrow over (r)}Pl is the location of the projected point in phase space.
In some examples, the set of parallel lines are projected onto a zero plane, which is a lower dimensional plane with a zero value for one axis. For example, in 3-dimensional phase space the zero plane may be a coordinate plane such as the X-Y Cartesian plane, the X-Z Cartesian plane, or the Y-Z Cartesian plane.
The zero plane may be defined as
P
o
≡r
0=0 with o={1,2,3} Eq. 12
where ro=0 indicates which coordinate is set to zero. For example, graph 902 corresponds to r1=0. The distance to propagate a line {right arrow over (R)}l with l∈{1,2, . . . , L} to the plane P is calculated as the intersection of a line R and one of the planes Po (composed by 2 perpendicular lines). Since the 3-dimensional case produces a determined system with 3 equations and 3 variables, the solution can be solved in vector form as:
t
o
l
=−n
o
l
·f
o Eq. 13
where tol is the propagation distance along line l in dimension o. Therefore, the projected points onto the plane Po are determined as:
{right arrow over (ρ)}Pl={right arrow over (n)}l+tol{circumflex over (f)} Eq. 14
where {right arrow over (ρ)}Pl is the location of the projected total phase point in zero plane Po.
As mentioned above, in some examples, the matrix of points may be rotated to reduce dimensionality. Additionally or alternatively, dimensionality reduction may comprise a change in the coordinate system. For example, as the matrix of points 800 comprises 3-dimensional points in a tilted 2-dimensional plane, a rotation may be applied to reduce the points to 2-dimensional points. As such, dimensionality reduction may be achieved by multiplying the coordinates {right arrow over (r)}Pl by the 3-dimensional rotation matrix composed by the coordinate system given by the normalized frequency vector {circumflex over (f)} and two perpendicular vectors. The perpendicular vectors are calculated by the vector product: {right arrow over (f)}1={circumflex over (f)}×{circumflex over (r)}i, {circumflex over (r)}k being a unitary vector, k∈{1,2,3}, and {right arrow over (f)}2={circumflex over (f)}×{right arrow over (f)}1. The rotation matrix can be expressed as Rot≡[{circumflex over (f)}, {circumflex over (f)}1, {circumflex over (f)}2] sorted either way in columns or vectors. Given this rotation matrix, the rotated points are then solved:
{right arrow over (ρ)}Pl={right arrow over (r)}PlRot Eq. 15
where {right arrow over (ρ)}pl is the location of the intersection point after rotation. As a result of the rotation, the dimension corresponding to the unitary vector {circumflex over (r)}k of the rotated projected point {right arrow over (ρ)}Pl is 0.
After obtaining the matrix of points, the points are grouped by plotting independent terms through the matrix. As described above, each independent term references groups of projected phase order points along parallel lines in the lower dimensional plane. As shown in
The slope of the parallel lines plotted through the matrix of points may be obtained in any suitable method. For example, a stochastic method may be used to find the most frequent slopes among the projected points. Given two points within the matrix of points, the slope is a ratio between the vertical and horizontal displacements. Likewise, for the set of points ρpl, the slope is
where αl is the slope, ρPl(2) is the coordinate in the first direction (e.g., vertical), ρPl(1) is the coordinate in the second direction (e.g., horizontal), and t indicates the matrix transpose. The resultant αl is a symmetric matrix in which the diagonal elements are infinite, as the divisor for these elements is zero. Applying an arctan operation creates a symmetric matrix which has diagonal elements of
θl=arctan(αl) Eq. 17
where θl is the set of angles. Then, the two most frequent angles are found from the mode of the off-diagonal elements, e.g., the upper triangular matrix. In some examples, the angles may be rounded to make the process less sensitive to computational precision. In other examples, any other suitable method may be used to obtain the two most frequent angles (e.g., binning). Further, the most frequent slopes may be obtained from the angles by applying a tangent operation. Once the most frequent slopes αkl with k∈{1,2} are found, the independent terms are calculated as
βkl=αkl·ρPl(1)−ρPl(2) Eq. 18
where βkl are the independent terms.
For example, the lines of
Assuming that {right arrow over (β)}k represents the set of unique independent terms for direction k with {right arrow over (β)}1={β11,β12 . . . ,β1l} and {β21,β22, . . . ,β2j}, the independent terms are sorted ascendingly which makes a table of I rows and J columns. In addition, as the slopes are parallel, the distance between independent terms is the same and there are two differences between consecutive independent terms Δk=|βk1−βk2|. Next, the terms are used to produce a matrix with dimensions (I,J) as:
where Δ1 is the difference between consecutive independent terms for {right arrow over (β)}1 and Δ2 is the difference between consecutive independent terms for {right arrow over (β)}2. The result of this operation are elements of a matrix that are related to the phase order.
The lines corresponding to the independent terms intersect at a point in
Once the phase order set {right arrow over (N)}, the matrix of points, and the look-up table are calculated, they may be stored and used in phase unwrapping. In some examples, the rotation matrix Rot, the line slopes αk, the independent terms {right arrow over (β)}k, and/or the distance between lines may also be stored.
In some examples, at 1306, the lower dimensional plane is normal to the reference phase vector. For example, the distance of the noisy phase point to the lower dimensional plane P can be calculated as
{tilde over (t)}=−{right arrow over ({tilde over (φ)})}·{circumflex over (f)}t Eq. 20
where {tilde over (t)} is the distance, {right arrow over ({tilde over (φ)})} is the noisy phase vector of the noisy phase point, and {circumflex over (f)}t is the transpose of the reference phase vector. Here, the tilde accent indicates a noisy measurement. In the case of a M×N array of pixels, the distance of a noisy phase point of the image array {right arrow over ({tilde over (φ)})}(m,n), can be calculated as
{tilde over (t)}(m,n)=−{right arrow over ({tilde over (φ)})}(m,n)·{circumflex over (f)}t Eq. 21
where {tilde over (t)}(m,n) represents the distance of the noisy phase point at pixel (m,n) of the image array. As such, the location of the projected noisy point is calculated as
{right arrow over ({tilde over (φ)})}P(m,n)={right arrow over ({tilde over (φ)})}(m,n)+{tilde over (t)}(m,n){circumflex over (f)} Eq. 22
where {tilde over (φ)}P(m,n) is the location within the lower dimensional plane P.
In such examples, the method may further comprise applying a rotation to the projected point to reduce dimensionality. In such examples, the rotation applied is that applied during generation of the matrix of points. For example, dimensionality may be reduced by using a rotation matrix:
{right arrow over ({tilde over (ϕ)})}(m,n)={right arrow over ({tilde over (φ)})}(m,n)·Rot Eq. 23
where Rot is the rotation matrix calculated and applied during the generation of the matrix of points (e.g., Eq. 12). As described above, the matrix Rot may be stored during and used during phase unwrapping.
Continuing with method 1300, in other examples, at 1310, the lower dimensional plane is a zero plane. In some such examples, when using 3 amplitude modulation frequencies (i.e., K=3), the zero plane may be the X-Y, X-Z, or Y-Z Cartesian plane.
Method 1300 further comprises, at 1312, comparing the projected point to a predetermined matrix of points, and determining independent terms.
{tilde over (β)}k(m,n)=αkl{tilde over (ϕ)}(1,m,n)−{tilde over (ϕ)}(2,m,n) Eq. 24
where {tilde over (ϕ)}(1, m,n) denotes the horizontal component of the phase of the location of the projected phase point of pixel (m,n), {tilde over (ϕ)}(2, m,n) is the vertical component of the location, αkl is the slop for direction k, and {tilde over (β)}k is the independent term for direction k.
As seen in
In other examples, the independent terms may be found using any other suitable method. For example, a dot product between the 2D location 1400 and a vector orthogonal to direction 1404 (or 1408) may be used to determine the first (or second) independent term. In another example, given an appropriate basis set, a change-of-basis matrix operation may be performed to project the noisy phase point onto a plane normal to the reference phase vector and obtain the independent terms.
Referring again to
where Δ1 is the distance between lines for the first set of parallel lines, Δ2 is the distance between lines for the second set of parallel lines, N is the lowest independent value of the first independent term, and β21 is the lowest independent value of the second independent term. The values i and j are then used to reference a look-up table (e.g., look-up table 1200) and retrieve the element (i,j), as described above with respect to
After selecting a most likely phase order set, method 1300 further comprises, at 1316, calculating a depth (or radial distance) value for the pixel. The most likely phase order set is added to the wrapped phase and the distance is calculated as a weighted sum, which may reduce jitter and improve precision. For the case of three modulation frequencies, the depth calculation is
where d is the depth value, {tilde over (φ)}0 is the wrapped phase of frequency o, nol is the most likely phase order for frequency o, and ξo is a normalized coefficient.
At 1504, method 1500 further comprises determining a location at which a projection line that passes through the set of noisy phase measurements in a K-dimensional phase space passes through a lower dimensional plane in the K-dimensional phase space, the projection line being parallel to a noise free phase evolution line. In some examples, at 1506, the projection line and the noise-free phase evolution line are normal to the lower dimensional plane. In some such examples, at 1508, method 1500 comprises performing a rotation on the predetermined matrix of points to reduce dimensionality. In other examples, at 1510, the lower dimensional plane is a zero plane. In some such examples, at 1512, K=3 and the zero plane is one of the X-Y Cartesian plane, the X-Z Cartesian plane, or the Y-Z Cartesian plane.
Continuing on
Method 1500 further comprises, at 1520, locating a corresponding set of noiseless phase orders by using the set of independent terms to reference a look-up table. As described above, in some examples, the elements of the look-up table may provide an index number of a phase order set within a series of phase order sets. Method 1500 further comprises, at 1522, determining a distance value for the pixel based upon the corresponding set of noiseless phase orders, and at 1524, outputting the distance value for the pixel.
Method 1600 further comprises, at 1606, projecting the plurality of parallel lines onto a lower dimensional plane in the K-dimensional space, thereby forming a matrix of points from intersections of the parallel lines with the lower dimensional plane. In some examples, at 1608, the lower dimensional plane is normal to the reference phase vector. In some such examples, at 1610, the method comprises rotating the matrix of points to reduce dimensionality. In other examples, at 1612, the lower dimensional plane is a zero plane and in some such examples, at 1614, K=3 and the zero plane is one of the X-Y Cartesian plane, the X-Z Cartesian plane, or the Y-Z Cartesian plane.
Method 1600 further comprises, at 1616, forming groups of points by plotting independent terms through the matrix of points. In some examples, at 1618, plotting independent terms through the matrix of points comprises plotting a first set of parallel lines corresponding to a first independent term and plotting a second set of parallel lines corresponding to a second independent term such that every point of the matrix of points is located at an intersection between a line of the first set of parallel lines and a line of the second set of parallel lines.
Method 1600 further comprises, at 1620, generating a look-up table that identifies a phase order set based on two or more independent terms. At 1622 the method comprises storing the matrix of points and the look-up table for use in phase unwrapping. In some examples, at 1624, the plurality of phase order sets, a rotation matrix, a plurality of slopes, and/or a value indicating the distance between independent terms is stored along with the matrix of points and the look-up table.
In some examples, at 1626, method 1600 further comprises performing phase unwrapping using the stored matrix of points and the look-up table to calculate a depth value for a pixel. As such, method 1600 may utilize method 1300 or method 1500.
The examples disclosed herein may achieve improved efficiency with fast computation and low memory usage compared to other phase unwrapping methods. Further, the disclosed examples may utilize fast operations on processors (e.g., multiplication and sorting) while avoiding interpolations. As such, the examples involve low-level computations that can be performed without a high-powered graphical processing unit (GPU).
In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
Computing system 1700 includes a logic machine 1702 and a storage machine 1704 storage machine 1704. Computing system 1700 may optionally include a display subsystem 1706 display subsystem 1706, input subsystem 1708, communication subsystem 1710, and/or other components not shown in
Logic machine 1702 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
Storage machine 1704 Storage machine 1704 includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 1704 may be transformed—e.g., to hold different data.
Storage machine 1704 may include removable and/or built-in devices. Storage machine 1704 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage machine 1704 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
It will be appreciated that storage machine 1704 includes one or more physical devices. However, aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
Aspects of logic machine 1702 and storage machine 1704 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 1700 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via logic machine 1702 executing instructions held by storage machine 1704. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
It will be appreciated that a “service”, as used herein, is an application program executable across multiple user sessions. A service may be available to one or more system components, programs, and/or other services. In some implementations, a service may run on one or more server-computing devices.
When included, display subsystem 1706 may be used to present a visual representation of data held by storage machine 1704. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 1706 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 1706 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 1702 and/or storage machine 1704 in a shared enclosure, or such display devices may be peripheral display devices.
When included, input subsystem 1708 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
When included, communication subsystem 1710 may be configured to communicatively couple computing system 1700 with one or more other computing devices. Communication subsystem 1710 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow computing system 1700 to send and/or receive messages to and/or from other devices via a network such as the Internet.
Another example provides a computing system, comprising a depth sensor comprising a plurality of pixels, each pixel configured to sense light, an illumination source configured to output light, a logic machine, and a storage machine holding instructions executable by the logic machine to control the illumination source to output amplitude-modulated light at K modulation frequencies, where K is an integer greater than or equal to two, and for each pixel of the plurality of pixels, make K phase measurements each at a different modulation frequency of the K modulation frequencies to form a set of noisy phase measurements, determine a location at which a projection line that passes through the set of noisy phase measurements in an K-dimensional phase space passes through a lower dimensional plane in the K-dimensional phase space, the projection line being parallel to a noise free phase evolution line, compare the location to a plurality of independent terms of a predetermined matrix of points in the lower dimensional plane, each point representing a corresponding predetermined noisy phase order set with known noise, and select a set of independent terms, locate a corresponding set of noiseless phase orders by using the set of independent terms to reference a look-up table, determine a distance value for the pixel based upon the corresponding set of noiseless phase orders, and output the distance value for the pixel. In some such examples, the projection line and the noise-free phase evolution line are normal to the lower dimensional plane. Additionally or alternatively, in some examples the instructions are further executable to perform a rotation on the predetermined matrix of points to reduce dimensionality. Additionally or alternatively, in some examples the lower dimensional plane comprises a zero plane. Additionally or alternatively, in some examples K=3 and the zero plane is one of the X-Y Cartesian plane, the X-Z Cartesian plane, or the Y-Z Cartesian plane. Additionally or alternatively, in some examples the predetermined matrix of points in the lower dimensional plane comprises a lattice formed by projecting a plurality of total phase points in a direction parallel to the noise-free phase evolution line onto the lower dimensional plane, each total phase point of the plurality of total phase points representing a total phase order set in K-dimensional phase space. Additionally or alternatively, in some examples the instructions are executable to determine each term of the plurality of independent term by projecting the location along a direction parallel to a set of lattice lines to intersect an axis in the lower dimensional plane, the set of lattice lines corresponding to the independent term.
Another example provides a method of generating a matrix of points and a look-up table for use in phase unwrapping, the method comprising generating a plurality of phase order sets from a reference phase vector in K-dimensional space by a stochastic process, the reference phase vector describing an evolution of phase with distance for K frequencies of amplitude modulated light, based on the plurality of phase order sets, generating a corresponding plurality of parallel lines in K-dimensional space, each parallel line having a point corresponding to a phase order set and a slope parallel to the reference phase vector, projecting the plurality of parallel lines onto a lower dimensional plane in the K-dimensional space, thereby forming a matrix of points from intersections of the parallel lines with the lower dimensional plane, plotting independent terms through the matrix of points, generating a look-up table that identifies a phase order set based on two or more independent terms, and storing the matrix of points and the look-up table for use in phase unwrapping. In some such examples, the method further comprises performing phase unwrapping by receiving a noisy phase point comprising K phase measurements from a time-of-flight pixel, determining a location at which a projection line that passes through the noisy phase point in K-dimensional phase space passes through the lower dimensional plane in the K-dimensional phase space, the projection line being parallel to the reference phase vector, comparing the location to the independent terms plotted through the matrix of points and selecting a set of independent terms, determining a phase order set by using the set of independent terms to reference the look-up table, determining a distance value based on the phase order set, and outputting the distance value for the time-of-flight pixel. Additionally or alternatively, in some examples, comparing the location to the independent terms comprises determining a first independent term by projecting the location along a first direction to intersect an axis, and determining a second independent term by projecting the location along a second direction to intersect the axis. Additionally or alternatively, in some examples the lower dimensional plane is normal to the reference phase vector. Additionally or alternatively, in some examples the method further comprises rotating the matrix of points to reduce dimensionality. Additionally or alternatively, in some examples the lower dimensional plane is a zero plane. Additionally or alternatively, in some examples K=3 and the zero plane is one of the X-Y Cartesian plane, the X-Z Cartesian plane, or the Y-Z Cartesian plane. Additionally or alternatively, in some examples plotting independent terms through the matrix of points comprises plotting a first set of parallel lines corresponding to a first independent term and plotting a second set of parallel lines corresponding to a second independent term such that every point of the matrix of points is located at an intersection between a line of the first set of parallel lines and a line of the second set of parallel lines. Additionally or alternatively, in some examples the method further comprises storing one or more of the plurality of phase order sets, a rotation matrix, a plurality of slopes corresponding to the independent terms, or a value indicating the distance between independent terms along with the matrix of points and the look-up table.
Another example provides a computing system, comprising a depth sensor comprising a plurality of pixels, each pixel configured to sense light, an illumination source configured to output light, a logic machine, and a storage machine holding instructions executable by the logic machine to control the illumination source to output amplitude-modulated light at K modulation frequencies, where K is an integer greater than or equal to two, and for each pixel of the plurality of pixels make K phase measurements each at a different modulation frequency of the K modulation frequencies to form a set of noisy phase measurements, determine a location at which a projection line that passes through the set of noisy phase measurements in a K-dimensional phase space passes through a zero plane in the K-dimensional phase space, the projection line being parallel to a noise-free phase evolution line, compare the location to a plurality of independent terms of a predetermined matrix of points in the zero plane, each point representing a corresponding predetermined noisy phase order set with known noise, and select a set of independent terms, locate a corresponding set of noiseless phase orders in a look-up table by using the set of independent terms to reference the look-up table, determine a distance value for the pixel based upon the corresponding set of noiseless phase orders, and output the distance value for the pixel. In some such examples, the predetermined matrix of points in the zero plane comprises a lattice formed by projecting a plurality of total phase points in a direction parallel to the noise-free phase evolution line onto the zero plane, each total phase point of the plurality of total phase points representing a total phase order set in K-dimensional phase space. Additionally or alternatively, in some examples K=3 and the zero plane is one of the X-Y Cartesian plane, the X-Z Cartesian plane, or the Y-Z Cartesian plane. Additionally or alternatively, in some examples the instructions are executable to determine each independent term of the plurality of independent terms by projecting the location along a direction parallel to a set of lattice lines to intersect an axis in the lower dimensional plane, the set of lattice lines corresponding to the independent term, and select the set of independent terms based on a location of axis intersection.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
