Patent Application
20120242839
The present disclosure relates generally to the field of driver assistance systems. The present disclosure more specifically relates to image sensor calibration systems and methods for a driver assistance system.
Driver assistance systems are becoming more and more popular for vehicles. Many driver assistance systems use one or more image sensors to capture images of the surrounding environment of the host vehicle in order to detect vehicles, pedestrians, obstacles, lanes, and traffic scenarios for warning the driver of potential dangers of collision, or activating automatic vehicle responses to avoid or mitigate collisions. To measure distances to detected objects, these image sensors have to be calibrated using a predefined target with known shape and size.
As such, there is a need to perform calibration of vehicle image sensors, so as to correctly detect and measure distances to all objects in the detection zone for accurate collision avoidance or mitigation.
According to one exemplary embodiment, a calibration apparatus includes a calibration target; a frame configured to hold the calibration target in place; a rotatable socket connected at one end to the frame; a stand which is connected to another end of the rotatable socket; a height adjusting portion connected at one end to the stand; and a platform connected to another end of the stand. The calibration target can be tilted and moved to a particular distance and orientation with respect to an image sensor to be calibrated by the calibration apparatus.
According to yet another exemplary embodiment, a calibration apparatus includes a movable platform; a calibration target held on the movable platform; a rotating unit configured to rotate the calibration target on the movable platform; and a height adjusting unit configured to adjust a height of the calibration target on the movable platform. The calibration target can be tilted and moved to a particular distance and orientation with respect to an image sensor to be calibrated by the calibration apparatus.
According to another aspect of the invention, an image calibration method includes capturing predetermined images used during calibration at a plurality of predefined positions and orientations with respect to an imaging system to be calibrated. The method also includes, based on the captured images at the plurality of predefined positions and orientations, forming a parameter set of images. The method further includes performing a calibration optimization process which includes obtaining a plurality of images at particular positions and orientations with respect to the imaging system to be calibrated. The method also includes using the parameter set of images to determine whether any of the images obtained during the calibration optimization process are to be categorized as outlier images and thereby removed.
The features, aspects, and advantages of the present invention will become apparent from the following description and accompanying exemplary embodiments shown in the drawings, which are briefly described below.
Some embodiments of the present invention are directed to a method for automatically measuring and calculating camera pitch, yaw, and roll angles. The method can be used such that image-ground transformation can be carried out correctly. This method can be used by a camera calibration system for a driver assistance system of a vehicle, according to an exemplary embodiment. Other embodiments of the present invention are directed to remove outlier images to improve the intrinsic parameters calibration accuracy.
In machine vision applications, because of lens distortion and imperfect lens mounting, camera calibration is an important step to determine the internal camera geometric and optical characteristics in order to extract accurate metric information from 2D images. Calibration is typically performed by observing and capturing images of a calibration object whose geometry in 3D space is known with very good precision, followed by a parameter estimation process using some calibration software. The most commonly used calibration object is a painted checkerboard with precisely known square size, and corner positions of each square in the image are used in the parameter estimation process.
Since the estimated principal point (optical center) of a camera varies from calibration to calibration, the stereo camera's pitch, yaw, and roll angles also change with each calibration. This variation causes problem in pitch sensitive applications such as lane departure warning, unless the correct pitch is measured or calculated after each calibration. A conventional method of calibrating the pitch, yaw, and roll angles involves measuring the ground truth of at least 4 ground points (downrange and cross range); ground point image capturing and rectification; feature point extraction from the rectified image of the ground points (currently manual); and calculation of the projection matrix and the 3 rotation angles.
One challenge is that the measurement (downrange, cross range) of the 4 ground points needs to be accurate, and a flat ground of 7 meters long in front of the camera is usually needed when the camera is installed on a vehicle. This, however, can not be easily done in a timely manner in a garage.
A method of calibrating the camera pitch, yaw, and roll angles according to an exemplary embodiment includes a mobile checkerboard and the usage of it in a specific way. This mobile checkerboard is a planar checkerboard that is mounted on a frame that can firmly hold it. The checkerboard frame is mounted on a metal socket through a universal joint that allows the frame to be able to freely move and stop at any orientation. The frame socket is mounted on a supporting stand with adjustable height, so that the elevation of the checkerboard can be adjusted in a predetermined range, e.g., a range of at least 500 mm vertically. The supporting stand is firmly mounted on a wheeled cart or platform, so that the checkerboard can be moved freely on the ground. The cart or platform is heavy enough to hold the framed checkerboard stably in motion.
According to an exemplary embodiment, the checkerboard elevation should be fixed at the same value throughout the entire calibration. The elevation should make the checkerboard totally inside the camera's vertical FOV at 2 meter range. The checkerboard images are to be captured at 3 (or more) predefined downranges. The checkerboard images are to be captured at multiple positions at each predefined range. The checkerboard images are to be captured at multiple orientations at each position and each range. An example of the checkerboard positions is described in
Since these checkerboard positions are aligned at the same elevation and predefined downrange, with the calibration software estimating these 3D positions in the optimization process, we can then calculate the pitch, yaw, and roll angles using the estimated parameters. Example of estimated checkerboard positions are shown in
This camera pitch, yaw, roll angle estimation method can be implemented in the calibration software so that it can be performed automatically once the ground truth is known.
Compared to conventional calibration methods, the embodiments of the present disclosure can be easy to implement in software and can be calculated automatically, whereby no extra steps are needed to obtain camera rotation angles once the calibration is done. This saves time and manpower for the total calibration effort.
Turning back to
The calibration apparatus 500 further includes a platform 560 connected to another end of the stand 540, to provide additional stability for the calibration apparatus 500. The top surface 560 of the platform is fixedly connected to the height adjusting portion 550, such as by screws, bolts, etc. The calibration apparatus also includes wheels 570A, 570B, to allow the calibration apparatus 500 to be easily moved to a particular distance from a device to be calibration, such as a vehicle having a stereo imaging system that needs to be calibrated.
The checkerboard 510 can be tilted and moved to a particular distance and orientation with respect to an image system to be calibrated by the calibration apparatus 500, whereby the tilting can be achieved by setting the rotatable socket 530 to a particular disposition, and whereby once it is set, the rotatable socket 530 stays in that particular position (e.g., a particular disposition of Body 1 and Body 2 of the rotable socket 530 of
Another aspect of the invention according to some embodiments is described below, with reference to FIGS. 1 and 7-10. In one exemplary embodiment, a method for calibration quality improvement is provided for a camera calibration system. The method identifies outlier images in stereo camera calibration and reduces errors in camera parameter estimation. This allows the system to further identify and remove outlier images and improve estimation accuracy of camera parameters. Further, this is easier to implement in software, or to perform interactively when the ground truth is unknown. This method is used by a camera calibration system for a driver assistance system of a vehicle, according to an exemplary embodiment.
In machine vision applications, because of lens distortion and imperfect lens mounting, camera calibration is a critical step to determine the internal camera geometric and optical characteristics in order to extract accurate metric information from 2D images. Calibration is typically performed by observing and capturing images of a calibration object whose geometry in 3D space is known with very good precision, followed by a parameter estimation process using some calibration software. The most commonly used calibration object is a painted checkerboard with precisely known square size, and corner positions of each square in the image are used in the parameter estimation process. Referring to
The quality of camera calibration, (i.e., the correctness and accuracy of camera parameter estimation) has a direct impact on determining the 3D position and size of an object based on its location and size in the image. This is especially true in stereo vision where far range accuracy is very sensitive to camera parameter errors. Because of image noise in the captured checkerboard images, extracted checker corner positions contain subpixel errors, which in turn result in errors in estimated camera parameters. Camera parameter estimation is normally done through a nonlinear optimization process, in which all intrinsic and extrinsic parameters (typically a few hundred of them) are all estimated simultaneously.
Because of the very high dimensionality of the parameter space and some of the parameters are correlated, a global minimum reached by the optimization process may not give the optimal estimate of the parameters, especially if one or more of the checkerboard images have non-Gaussian noises in its extracted corner positions, or the checkerboard is in an image area that has extra distortion not modeled by the lens distortion model. These “outlier” images tend to negatively affect the optimization process, and contribute errors to the estimated parameters. The challenge that is addressed in this another exemplary embodiment is how to identify these “outlier” images in the parameter estimation process.
A conventional method of identifying “outlier” images is by observing the reprojection error distribution after the optimization is done. Reprojection errors are calculated as the difference between measured corner positions and projected corner positions based on estimated parameters. A plot showing a typical reprojection error distribution is shown in
A problem of the conventional method is that “outlier” images could still exist in the remaining images with small reprojection errors, since the reprojection errors are calculated after the optimization is done which minimizes the reprojection errors. In other words, two parameters that have errors in opposite directions could still make the reprojection errors small.
A different approach for identifying the “outlier” images in the calibration process, as adopted in the another exemplary embodiment, is to utilize some constraints/knowledge on the 3D positions and orientations of the checkerboard poses in the captured images, besides reprojection errors, in outlier identification. The checkerboard images are captured at predefined positions and orientations, which serve as a kind of ground truth. As such, the checkerboard positions and orientations become part of the parameter set and are estimated in the optimization process. That way, after optimization, any checkerboard image with its estimated position and/or orientation far from the ground truth can be identified as an outlier and can be removed. An example of the “known” checkerboard positions is provided in
An example of estimated checkerboard positions are shown in
The present disclosure has been described with reference to example embodiments, however persons skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the disclosed subject matter. For example, although different example embodiments may have been described as including one or more features providing one or more benefits, it is contemplated that the described features may be interchanged with one another or alternatively be combined with one another in the described example embodiments or in other alternative embodiments. Because the technology of the present disclosure is relatively complex, not all changes in the technology are foreseeable. The present disclosure described with reference to the exemplary embodiments is manifestly intended to be as broad as possible. For example, unless specifically otherwise noted, the exemplary embodiments reciting a single particular element also encompass a plurality of such particular elements.
Exemplary embodiments may include program products comprising computer or machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Exemplary embodiments illustrated in the methods of the figures may be controlled by program products comprising computer or machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such computer or machine-readable media can be any available media which can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such computer or machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of computer or machine-readable media. Computer or machine-executable instructions comprise, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions. Software implementations of the present invention could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps.
It is also important to note that the construction and arrangement of the elements of the system as shown in the preferred and other exemplary embodiments is illustrative only. Although only a certain number of embodiments have been described in detail in this disclosure, those skilled in the art who review this disclosure will readily appreciate that many modifications are possible (e.g., variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters, mounting arrangements, use of materials, colors, orientations, etc.) without materially departing from the novel teachings and advantages of the subject matter recited. For example, elements shown as integrally formed may be constructed of multiple parts or elements shown as multiple parts may be integrally formed, the operation of the assemblies may be reversed or otherwise varied, the length or width of the structures and/or members or connectors or other elements of the system may be varied, the nature or number of adjustment or attachment positions provided between the elements may be varied. It should be noted that the elements and/or assemblies of the system may be constructed from any of a wide variety of materials that provide sufficient strength or durability. Accordingly, all such modifications are intended to be included within the scope of the present disclosure. The order or sequence of any process or method steps may be varied or re-sequenced according to alternative embodiments. Other substitutions, modifications, changes and omissions may be made in the design, operating conditions and arrangement of the preferred and other exemplary embodiments without departing from the spirit of the present subject matter.
This application claims priority from Provisional Application 61/467,827; filed Mar. 25, 2011, incorporated herein by reference in its entirety. This application also claims priority from Provisional Application 61/467,863; filed Mar. 25, 2011, incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61467863 | Mar 2011 | US | |
| 61467827 | Mar 2011 | US |
