The utility of computer vision systems in a variety of applications is recognized. For example, autonomous navigation systems (e.g., for vehicles and robots) rely heavily on such systems for obstacle detection and navigation in surrounding environments. Such systems enable the navigation and/or surveillance of difficult or dangerous terrain without putting human operators at risk.
However, most existing systems for autonomous navigation lack versatility in that they are typically environment-specific. For example, GPS-based navigation systems work well in outdoor environments, but perform poorly indoors. Navigation systems that rely on information from wheel encoders work well when implemented in ground vehicles, but are unsuitable for use in, say, aerial vehicles. Moreover, most existing systems that operate by analyzing video or image data can provide knowledge of past motion, but cannot provide timely (e.g., real time) knowledge of current motion and/or position.
Therefore, there is a need in the art for a method and apparatus for visual odometry that is environment-independent and can reliably provide motion and/or position estimates in substantially real time.
A method and apparatus for visual odometry (e.g., for navigating a surrounding environment) is disclosed. In one embodiment a sequence of scene imagery is received (e.g., from a video camera or a stereo head) that represents at least a portion of the surrounding environment. The sequence of scene imagery is processed (e.g., in accordance with video processing techniques) to derive an estimate of a pose relative to the surrounding environment. This estimate may be further supplemented with data from other sensors, such as a global positioning system or inertial or mechanical sensors.
So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.
The present invention discloses a method and apparatus for visual odometry (e.g., for autonomous navigation of moving objects such as autonomous vehicles or robots). Unlike conventional autonomous navigation systems, in one embodiment, the present invention relies primarily video data to derive estimates of object position and movement. Thus, autonomous navigation in accordance with the present invention is substantially environment-independent. Environment-specific sensors, such as those conventionally used in autonomous navigation systems, serve mainly as optional means for obtaining data to supplement a video-based estimate.
In step 106, the method 100 processes the sequence of scene imagery to derive a position estimate therefrom. That is, the method 100 estimates a current position of the object requiring navigation directly from the received sequence of scene imagery. In one embodiment, the sequence of scene imagery is processed in accordance with any suitable known method for video processing.
Once a position estimate has been derived from the sequence of scene imagery, the method 100 optionally proceeds to step 108 and supplements the position estimate with additional data. Thus, in this embodiment, the video-based position estimate derived in step 106 may be considered a preliminary estimate that is subsequently refined by incorporating data from other sources. In one embodiment, this additional data includes data provided by at least one additional sensor, such as at least one of: a GPS system, inertial sensors and mechanical sensors (e.g., wheel encoders).
Once a position estimate has been derived (with or without the additional data), the method 100 terminates in step 110.
The method 100 thereby enables rapid, accurate motion and position estimation that is independent of the environment in which the method 100 functions. Because the method 100 relies primarily (and in some cases exclusively) on video data to derive motion and position estimates, it can be implemented to advantage in virtually any location: outdoors, indoors, on the ground, in the air, etc.
The method 200 is initialized at step 202 and proceeds to step 204, where the method 200 locates point features in a current frame of the sequence of scene imagery. In one embodiment, the located point features are features that are expected to remain relatively stable under small to moderate image distortions. For example, in one embodiment, the point features are Harris corners, as described by C. Harris and M. Stephens in A Combined Corner and Edge Detector (Proc. Fourth Alvey Vision Conference, pp. 147-151, 1988). Point features can be any identifiable element of the frame that can be reliably tracked. In one embodiment, up to hundreds of point features are located in step 202. For example,
In step 206, the method 200 tracks the point features located in step 204 over a plurality of subsequent frames (e.g., by matching the features to corresponding features in the subsequent frames). In one embodiment, the point features are tracked for as long as the features remain in the field of view. In one embodiment, tracking is performed without any geometric constraints.
In step 208, the method 200 produces a set of trajectories based on the feature tracking data obtained in step 206. The trajectories represent changes in the location and/or orientation of the tracked features relative to the object requiring navigation over time. In one embodiment, matched features are essentially linked between frames. Referring again to
Once the set of trajectories has been established, the method 200 proceeds to step 210 and generates a plurality of incremental frame-to-frame pose estimates for the vehicle, robot or the like that requires navigation, based on the information conveyed by the point feature trajectories. In one embodiment, “pose” is estimated with six degrees of freedom and is defined as three-dimensional (e.g., in x, y, z coordinates) location plus angular orientation. In one embodiment, pose estimates are generated in accordance with a geometric estimation method. Geometric estimation methods for generating pose estimates may vary depending on the means for capturing the original sequence of scene imagery (e.g., monocular video input, stereo input, etc.).
In step 212, the method 200 evaluates the pose estimates and selects the most likely estimate to be indicative of the current pose. In one embodiment, evaluation of pose estimates is performed in accordance with a known random sample consensus (RANSAC) technique (e.g., as discussed by D. Nister in Preemptive RANSAC for Live Structure and Motion Estimation, IEEE International Conference on Computer Vision, pp. 199-206, 2003), as discussed in greater detail below.
The method 200 terminates in step 214.
In step 306, the method 300 computes the strength, s, of the frame's corner response. For example, in one embodiment a Harris corner detector computes the locally averaged moment matrix computed from the image gradients. The eigenvalues of the moment matrix are then combined to compute a corner response or “strength”, the maximum values of which indicate corner positions.
In one embodiment designed for very efficient computation (e.g., using a general purpose computer), s is computed as follows: for every output line of corner response, temporary filter outputs are needed for a certain number of lines above and below the current output line. All filter outputs are computed only once and stored in wrap-around buffers for optimal cache performance. The wrap-around buffers represent the temporary filter outputs in a rolling window. The rolling window contains the minimal number of lines necessary in order to avoid recomputing any filter outputs.
For example, in one embodiment, the horizontal and vertical derivatives of the image frame are represented as Ix and Iy, respectively. In one embodiment, Ix and Iy are computed by horizontal and vertical filters of the type [−1 0 1] and shifted down one bit before performing multiplications to keep the input down to eight bits and output down to sixteen bits.
In this case, the wrap-around buffers and resulting corner responses are updated line-by-line using four “sweeps” per line. The first sweep updates the wrap-around buffers for IxIx, IxIy and IyIy. In one embodiment, the wrap-around buffers for IxIx, IxIy and IyIy are fice lines long, and the typical sweep updates one line, positioned two lines ahead of the current output line of corner response.
The second sweep convolves all lines in the wrap-around buffers vertically with a binomial filter (e.g., [1 4 6 4 1]) in order to produce three single lines of thirty-tow-bit filter output: gxx, gxy and gyy. In one embodiment, this is accomplished by shifts and additions to avoid expensive multiplications.
The third sweep convolves horizontally with the same binomial filter used in the second sweep to produce the thirty-two-bit single lines: Gxx, Gxy, Gxy. Gxx, Gxy, Gxy are stored back in the same place as gxx, gxy and gyy, but are shifted two pixels.
Finally, the fourth sweep computes, in floating point:
the determinant, d=GxxGyy−GxyGxy (EQN. 1)
the trace, t=Gxx+Gyy (EQN. 2)
and the strength, s=d−kt2 (EQN. 3)
of the corner response (where k=0.06). In one embodiment, the first through fourth sweeps are all implemented in multimedia extension (MMX) chunks of 128 pixels and interleaved manually to avoid stalls and to make optimal use of both pipelines.
Referring back to
The method 300 terminates in step 312.
The next phase of the method 400 establishes frame-to-frame feature correspondence. This frame-to-frame feature correspondence can be established in accordance with a variety of known methods, including optical flow and area correlation techniques. Steps 406-408 illustrate one exemplary process for establishing frame-to-frame feature correspondence, which is optimized for speed of computation (e.g., on a general purpose computer).
In step 406, the method 400 evaluates potential point feature matches between the first and second frame using normalized correlation. In one embodiment, normalized correlation is performed over an eleven-pixel-by-eleven-pixel window centered on the detected point feature. In one embodiment, uniform weighting is used across the whole window for speed. Each window is copied from the image frame and laid out consecutively in memory as an n=121 byte vector (in one embodiment padded to 128 bytes for convenience). For each window, the following values are pre-computed:
Then, for each potential match, the following scalar product is computed between the two windows:
D=Σ I1I2 (EQN. 7)
The normalized correlation is then:
(nD−A1A2)C1C2 (EQN. 8)
Once the normalized correlations are computed for each potential match, the method 400 proceeds to step 408 and determines which matches to accept, in accordance with mutual consistency. In accordance with this theory, every point feature in the first image frame is involved in a number of normalized correlations with point features from the second image frame (e.g., as determined by the maximum disparity). The point feature from the second image frame that produces the highest normalized correlation is thus selected as the preferred match to the point feature in the first frame. Conversely, each point feature in the second image frame will also generate a preferred match in the first image frame. Accordingly, pairs of point features that mutually designate each other as the preferred match are accepted as valid matches. As described above, this matching technique may be performed over a plurality of image frames in order to generate a trajectory that illustrates the motion of a point feature over time.
The method 400 terminates in step 410.
The method 500 may be implemented, for example, in accordance with step 210 of the method 200 discussed above. In particular, the method 500 is useful in generating frame-to-frame incremental pose estimates based on monocular video input (e.g., data from a single moving video camera).
The method 500 is initialized at step 502 and proceeds to step 504, where the method 500 receives a plurality of point feature trajectories for point features tracked through a plurality of frames of the sequence of scene imagery (e.g., received as a feed from a single video camera mounted to a moving object requiring navigation). In step 506, the method 500 estimates, based on the received trajectory data, the poses of the object requiring navigation relative to the identified point features from among the plurality of frames.
In one embodiment, pose estimation in accordance with step 506 is performed in accordance with a five-point algorithm (e.g., as described in U.S. patent application Ser. No. 10/798,726, filed Mar. 11, 2004, which is herein incorporated by reference in its entirety) and pre-emptive RANSAC, followed by an iterative refinement. Thus, the method 500 generates a set of possible pose solutions or hypotheses based on the provided point features. These hypotheses are generated by selecting a subset of the available point feature trajectories. In one embodiment, this subset includes at least five randomly selected point feature trajectories. Each of these hypotheses is then evaluated against all available point feature trajectories to determine which hypothesis is consistent with the maximum number of feature trajectories. In one embodiment, this maximally consistent hypothesis is taken to be the most likely to be correct.
In step 508, the method 500 uses the estimated pose determined in step 506 to triangulate the observed point feature trajectories into a plurality of three-dimensional (3D) points. In one embodiment, triangulation is performed using the first and last observed point features along the point feature trajectory. In further embodiments, triangulation is performed in accordance with optimal triangulation according to directional error. In one embodiment, if it is not the first time that step 508 is being executed, a scale factor between the present point feature trajectory results and an immediately previous point feature trajectory result is estimated (e.g., in accordance with a preemptive RANSAC procedure). The present point feature trajectory results then replace the previous results.
In step 510, the method 500 receives additional point feature trajectory data, e.g., in the form of a stream of video input as the associated point features are tracked for a number of subsequent frames (e.g., subsequent to the point at which the point feature trajectories were first received in step 504). In step 512, the method 500 computes, based on the additional point feature trajectory data, the current pose with respect to the known 3D points (e.g., as established in step 508). In one embodiment, pose estimation is performed in accordance with a three-point, two-dimensional-to-three-dimensional algorithm and pre-emptive RANSAC, followed by an iterative refinement. One known three-point algorithm (described, for example, by R. Haralick, C. Lee, K. Ottenberg and M. Nolle in Review and Analysis of Solutions of the Three Point Perspective Pose Estimation Problem, International Journal of Computer Vision, 13(3):331-356, 1994, and by various textbooks) uses the correspondence of three two-dimensional image points to three three-dimensional world points to estimate the camera pose.
Following pose estimation with respect to the known three-dimensional points in step 512, the method 500 proceeds to step 514 and re-triangulates additional 3D points with relation to the new point feature trajectory data. In one embodiment, re-triangulation is performed using the first and last observed feature points along the trajectory (e.g., which now includes the new feature point trajectory data). The method 500 then proceeds to step 516 and determines whether tracking should be continued (e.g., whether additional point feature trajectory data should be processed) from step 510. In one embodiment, the determination as to whether to continue with further iterations from step 510 may be made in accordance with any one or more of a number of application-specific criteria, such as at least one of: computational cost and environmental complexity. For example, the three-point pose estimation technique discussed above is generally less computationally complex than other related methods, so performing additional three-point estimates relative to the number of five-point estimates will typically decrease overall computational load. However, the accuracy of the three-point pose estimation technique depends directly on the accuracy of the triangulated three-dimensional points, which may be subject to errors, especially in complex scene environments. Thus, balancing these considerations on an application-by-application method id generally desirable to determine the optimal number of iterations of steps 510-514 for a given application. In one embodiment, however, the number of iterations from step 510 is pre-set to three.
If the method 500 determines in step 516 that tracking should be continued from step 510, the method 500 returns to step 510 and proceeds as described above. Alternatively, if the method 500 determines in step 516 that tracking should not be continued from step 510, the method 500 proceeds to step 518.
In step 518, the method 500 determines whether tracking should be continued (e.g., whether additional feature trajectory data should be processed) from step 504. In one embodiment, processing continues from step 504 for a number of iterations, where the number of iterations depends on application-specific criteria such as the motion speed and probability of pose and/or triangulation errors. In one embodiment, the number of iterations performed from step 504 is pre-set to three. If the method 500 determines in step 518 that tracking should be continued from step 504, the method 500 returns to step 504 and proceeds as described above. Alternatively, if the method 500 determines in step 518 that tracking should not be continued from step 504, the method 500 proceeds to step 520.
In step 520, the method 500 inserts a firewall into the stream of input data such that future triangulations of 3D points will not be performed using observations that precede the most recent firewall. Thus, for the purposes of triangulation, the frame of the sequence of scene imagery immediately following the firewall is considered the first frame. In other words, the three-dimensional points used for preceding iterations are discarded and a completely new set of three-dimensional points is estimated. This helps to reduce the propagation of errors (e.g., in 3D points positioning, pose estimation, etc.) throughout execution of the method 500. The method 500 then returns to step 504 and proceeds as described above.
The method 600 is initialized at step 602 and proceeds to step 604, where the method 600 receives point feature trajectories (e.g., as embodied in individual feeds from two moving video cameras mounted to a moving vehicle or robot). The point feature trajectories are received from two different views that present different perspectives of the same point feature trajectories (e.g., as viewed from a left video camera and a right video camera). The method 600 then proceeds to step 606 and matches point features between the two views as presented in incoming images or sequences of scene imagery.
In step 608, the method 600 triangulates the matches established in step 606 into 3D points using knowledge of stereo calibration data. Additional point feature trajectory data is then received in step 610.
In step 612, the method 600 estimates, based on the received point feature trajectory data, the relative poses of the object requiring navigation (e.g., upon which the stereo head is mounted) among a plurality of frames of the sequences of scene imagery. In one embodiment, pose estimation in accordance with step 612 is performed in accordance with a three-point algorithm (e.g., as discussed above, using features from, for example, the left images) and pre-emptive RANSAC, followed by an iterative refinement based on features in both the left and right images. Thus, the method 600 generates a set of possible pose solutions or hypotheses based on the provided feature points. These hypotheses are generated by selecting a subset of the available feature trajectories. In one embodiment, this subset includes at least three randomly selected feature trajectories. Each of these hypotheses is then evaluated against all available feature trajectories to determine which hypothesis is the most likely to be correct (e.g., based on maximal consistency with all features).
In step 614, the method 600 determines whether tracking should be continued (e.g., whether additional point feature trajectory data should be processed) from step 610. As discussed above, this determination may be made based on application-specific criteria, or iterations may be performed a fixed number of times. If the method 600 determines in step 614 that tracking should be continued from step 610, the method 600 returns to step 610 and proceeds as described above. Alternatively, if the method 600 determines in step 614 that tracking should not be continued from step 610, the method 600 proceeds to step 616.
In step 616, the method 600 triangulates all new point feature matches in accordance with observations in the left and right images. The method 600 then proceeds to step 618 and determines whether tracking should be continued from step 610. As discussed above, this determination may be made based on application-specific criteria, or iterations may be performed a fixed number of times. If the method 600 determines in step 618 that tracking should be continued from step 610, the method 600 returns to step 610 and proceeds as described above. Alternatively, if the method 600 determines in step 618 that tracking should not be continued from step 610, the method 600 proceeds to step 620.
In step 620, the method 600 discards all existing 3D points and re-triangulates all 3D points based on the new point feature trajectory data and accordingly inserts a firewall into the stream of input data such that future triangulations of 3D points will not be performed using observations that precede the most recent firewall. The method 600 then returns to step 610 and proceeds as described above.
Alternatively, the visual odometry module 705 can be represented by one or more software applications (or even a combination of software and hardware, e.g., using Application Specific Integrated Circuits (ASIC)), where the software is loaded from a storage medium (e.g., I/O devices 706) and operated by the processor 702 in the memory 704 of the general purpose computing device 700. Thus, in one embodiment, the visual odometry module 705 for estimating motion and position described herein with reference to the preceding Figures can be stored on a computer readable medium or carrier (e.g., RAM, magnetic or optical drive or diskette, and the like).
In further embodiments, the present invention can be implemented in an integrated sensing device that combines visual odometry with conventional navigation devices (such as GPS, inertial measurement units, compasses and the like). In such embodiments, the six degrees of freedom motion estimates produced by visual odometry are used to correct estimates produced by conventional sensors, or vice versa. This integrated system can thus produce a single navigation solution incorporating all available sensor inputs. An advantage of such a system over conventional devices is that an integrated navigation system can operate either on visual input alone or on visual input supplemented with additional sensor input for more accurate and stable localization.
Thus, the present invention represents a significant advancement in the field of autonomous navigation. A method and apparatus are provided that enable a moving object (e.g., an autonomous vehicle or robot) to navigate a surrounding environment regardless of the nature of the surrounding environment. By processing primarily video data, which is obtainable in substantially any environment or conditions, location and movement can be accurately estimated. Data from additional, environment-specific sensors, such as those conventionally used in autonomous navigation systems, may then be optionally used to supplement estimates derived from the video data.
While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
This application claims benefit of U.S. provisional patent application Ser. No. 60/581,867, filed Jun. 22, 2004, which is herein incorporated by reference in its entirety.
The invention was made with Government support under grant number MDA972-01-9-0016 awarded by the Defense Advanced Research Projects Agency (DARPA). The Government has certain rights in this invention.
Number | Date | Country | |
---|---|---|---|
60581867 | Jun 2004 | US |