1. Field of the Invention
The invention relates to vision systems and, more particularly, the present invention relates to a method and apparatus for detecting obstacles using a vehicular-based vision system.
2. Description of the Background Art
Vehicular vision systems generally comprise a camera (or other sensor) mounted to a vehicle. An image processor processes the imagery from the camera to identify obstacles that may impede the movement of the vehicle. To identify obstacles, a plane is used to model the roadway in front of the vehicle and the image processor renders obstacles as point clouds that extend out of the plane of the roadway. By using such a planar model, the processing of imagery from “on-road” applications of vehicular vision systems is rather simple. The image-processing system must recognize when the point cloud is extending from the roadway plane and deem the point cloud simply to be an obstacle to be avoided.
In “off-road” applications, where the ground upon which the vehicle is to traverse is non-planar, the terrain cannot be modeled as a simple plane. Some applications have attempted to model the off-road terrain as a plurality of interconnecting planes. However, such models are generally inaccurate and cause the vehicle to identify obstacles that could, in reality, be traversed by the vehicle. As such, unnecessary evasive action is taken by the vehicle.
Therefore, there is a need for a method and apparatus of performing improved obstacle detection that is especially useful in “off-road” applications.
The invention provides a method and apparatus for detecting obstacles in non-uniform environments, e.g., an off-road terrain application. The apparatus uses a stereo camera and specific image-processing techniques to enable the vehicle's vision system to identify drivable terrain in front of the vehicle. The method uses the concept of a non-drivable residual (NDR), where the NDR is zero for all terrain that can be easily traversed by the vehicle and is greater than zero for terrain that may not be traversable by the vehicle. The method utilizes a depth map having a point cloud that represents the depth to objects within the field of view of the stereo cameras. The depth map is organized into small tiles; each tile is represented by the average of the point cloud data contained within. The method scans columns of pixels in the image to find sequences of “good” points that are connected by line segments having an acceptable slope. Points that lie outside of the acceptable slope range will have an NDR that is greater than zero. From this information regarding obstacles and the terrain before the vehicle, the vehicle control system can accurately make decisions as to the trajectory of the vehicle.
So that the manner in which the above recited features of the present invention are attained and can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to the embodiments thereof which are illustrated in the appended drawings.
It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.
The method does not recognize specific objects but labels areas that are difficult or impossible to traverse. Also, the method does not determine if an area on the other side of an obstacle can be reached, that process is left to the route planner that is responsible for that task.
An advantage of the non-drivable residual (NDR) method of the present invention is that it enables the evaluation of the change in vertical height from one place to another relative to the range of heights that would occur for drivable slopes. As such, the method uses mobility constraints for a particular vehicle and compares the constraints to the slope of the terrain proximate the vehicle. As illustrated in
At step 410, the method 400 determines whether the current point is within the drivable slope of the last point. If the answer is negative, the method 400 proceeds to step 412 and determines if the current point is within the drivable slope of the last good point. If the current point is within the drivable slope, or if the current point was determined in step 410 to be within the drivable slope of the last point, at step 414, the NDR for the current point is set to 0, and the last good point is updated to the current point. Then, at step 416, the zero NDR for the current point is stored. At step 418, the method 400 determines whether there is more,data to be processed. If there is more data, the last point is updated to the current point at step 420 and a loop is made back to step 408 for the selection of a new, current point.
However, if during step 412 a determination is made that the current point is not within the drivable slope of the last good point, a non-zero NDR with respect to the last good point is calculated for the current point at step 424. At step 416, the non-zero NDR is then stored for the current point. At step 418, the method queries whether more data is to be processed. If the query is affirmatively answered, the method 400 proceeds to step 420 and sets the current point to the last point and proceeds to step 408 to process the next point
When a determination is made in step 418 that there is no more data to be processed, the method 400 proceeds to step 426 wherein the points and NDRs are projected onto a map. At step 428, the map is used to plan a route that will avoid any detected obstacle. The plan may then be executed. For example, the map contains a two-dimensional matrix of values where zero value and low values represent passable terrain (i.e., terrain that does not exceed the mobility constraint of the vehicle) and high values represent impassable terrain (i.e., terrain that exceeds the mobility constraint of the vehicle). The specific thresholds assigned to produce “low” and “high” indications are defined by the particular vehicle that is traversing the terrain. Consequently, the map identifies regions in which the mobility constraints of the particular vehicle are exceeded and not exceeded.
The following calculation is applied to pixels (points) in one column of the image at a time. If there is a stereo dropout (unavailable data), the computation, continues with the next available pixel. The only state variables are the last point and the last good point. As mentioned above, the points may be processed simultaneously in rows and further comparitive processing can be performed to ensure accuracy of the computations.
Let (X,Y,Z) be the world coordinates of the point imaged at pixel (x,y) in the image. Assume that the world coordinates have been suitably transformed so that the Y axis is vertical. In practice, this transformation is achieved with input from an inertial navigation system (INS) which relates the camera pose to the world system. (In the usual system, X points right, Y points down, and Z points forward.) Let (X,Y,Z)L be the coordinates of the last point, and (X,Y,Z)G be the coordinates of the last “good” point. The initial values of these points are:
(X,Y,Z)L=(X,Y,Z)G=(0,−h,0)
where h is the camera height.
To compute the non-drivable residual (NDR or Rnd) for point (X,Y,Z), first compute the displacement from the last point:
(ΔX,ΔY,ΔZ)L=(X,Y,Z)−(X,Y,Z)L
The distance traveled (projected onto the XZ plane) is
dL={square root}{square root over (ΔXL2+ΔZL2)}
Let sdi be the maximum slope of a drivable incline (uphill or downhill). The limiting values for a drivable ΔY are:
ΔYuphill=−sdidL and ΔYdownhill=sdidL
If ΔYuphill≦ΔY≦ΔYdownhill, then the method has found a nominally flat, level place. Set Rnd=0 and update the last point and the last good points:
(X,Y,Z)L←(X,Y,Z) and (X,Y,Z)G←(X,Y,Z).
Otherwise, the change in elevation indicates a possible obstacle. To measure the severity of the height change, first the method computes the distance from the last good point:
dG={square root}{square root over (ΔXG2+ΔZG2)} where (ΔX,ΔY,ΔZ)G=(X,Y,Z)−(X,Y,Z)G
The ΔY limits for computing the residual are:
ΔYuphill=−sdidG and ΔYdownhill=sdidG
The residual is given by:
The residual is compared to a pre-defined threshold. If the residual is greater than the threshold, then the potential obstacle is deemed an actual obstacle to be avoided, i.e., the terrain is not traversable. Lastly, the method always updates the last point: (X,Y,Z)L←(X,Y,Z) and, if Rnd=0, then the method also updates the last good point: (X,Y,Z)G←(X,Y,Z).
Spurious values in the obstacle map can be suppressed by applying the method to average values of (X,Y,Z). Most of the experiments and tests have been done with averages computed for non-overlapping blocks of 5×5 pixels. Good results have also been obtained for overlapping, variable, sized patches ranging from 40 pixels square in the foreground to a minimum of 8 pixels square at row 68 out of 320. The main issue with the larger, overlapping averages is the increase in computation time. To obtain average values of (X,Y,Z), the quantity (1/Z) is approximated by a linear function of the pixel coordinates (x, y) in the patch. The value of (1/Z) obtained from the fit is used to compute Z at the center of the patch. X and Y are then computed from Z, the pixel coordinates, and the camera center and focal length.
The average is computed in camera coordinates, and then transformed to world coordinates. The transformation matrix includes the camera-to-vehicle rotation obtained from camera calibration, and the vehicle-to-world transformation obtained from the vehicle pose sensors.
While foregoing is directed to various embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
This invention was made with U.S. government support under contract number MDA972-01-9-0016. The U.S. government has certain rights in this invention.