Patent Application
20240273762
The invention relates to the field of technical vision, in particular to methods for calibrating the external parameters of video cameras.
Operation of video cameras while as part of land vehicles (LV), including autonomous ones, implies special operating conditions. Vibration and mechanical stress during the vehicle movement lead to changes in the external parameters of the video camera, such as rotation angle (pitch, roll and yaw), which negatively affects the parameters of the rotation matrix that converts the coordinates of the observed object from the video camera coordinate system into the LV coordinate system.
Thus, the uncorrected change of camera rotation angles during movement leads to the accumulation of errors in localization of the observed object or autonomous LV and decreases the accuracy of localization of these objects.
The following methods of calibrating the external parameters of video cameras are known in the pertinent art.
A method is known for automatic calibration of a video camera based on a mobile platform (KR20150096128), which includes the following steps: obtaining four or more images of the target object in different directions from a video camera located on a mobile platform, a preprocessing stage, selecting several straight lines from each input image, obtaining the midline by classifying the types of detected straight lines, determining the coordinates of the vanishing point based on the specified midline, and the stage of obtaining the parameter value of applying the coordinates of the detected vanishing point to the calibration formula to calculate the values of internal parameters.
A method is known for online calibration of a video system (US2011115912) using vanishing points estimated from frames of images from a video camera containing identified markings or road edges. Vanishing points are determined by finding or extrapolating at least one left and/or right side of the road markings or edges to the intersection point, whereby the long-term average location of the vanishing point is calculated using temporal filtering methods from a sequence of images, even when only one side of the road, lane marking or edge is visible in any given frame at a given time, and the yaw and pitch angles of the video camera are derived from the time-averaged coordinates of the vanishing point location.
A common disadvantage for the presented technical solutions is the use of multiple images obtained from video cameras to determine the vanishing point, which reduces the speed of image processing and increases the required computing power.
A method is known for determining the roll angle of a video camera mounted on a vehicle (CN112017249), which includes the following steps: obtaining images from a vehicle-mounted video camera, extracting all straight lines in the horizontal direction of the area of the vehicle body in the image, determining all slopes of straight lines in the horizontal direction and identifying among them the average value of the slope, calculating the video camera roll angle in accordance with the identified average value of the slope. The azimuth and pitch angle are obtained from the position of the vanishing point, determined by the road lane lines in the image.
In this method the vanishing point position is preferably determined using two selected straight lines in the image, which leads to decreased accuracy of determining vanishing point coordinates and, as a result, to lower accuracy of calibration of video cameras external parameters.
A method is known for automatic calibration of video cameras (US2018324415), according to which, during the movement of a vehicle, images of environment are received from video cameras, key image points are selected in the area limited by the location of the road, while key points are tracked using the optical flow method, filtration procedures are applied to key points, at least two straight lines are determined that correspond to opposite sides of the road, then the vanishing point is determined for the selected lines, the position of the obtained vanishing point is compared with the position of the principal point; based on the deviation obtained, the values of the pitch and yaw angles of the video camera are determined using the following formulas:
The presented method is characterized by the fact that straight lines are determined in several images by constructing a straight line (trajectory) of movement of the selected point from one image to another, which reduces the speed of image processing and increases the required computing power of the on-board computer.
The described technical solution is closest in technical essence to the claimed invention and can act as a prototype.
The claimed invention is designed to create a method for calibrating the external parameters of video cameras used in land vehicles by determining the vanishing point position using selected rectilinear segments in a single image, obtained from a video camera, which allows for a high accuracy assessment of changes in the video cameras rotation angles (pitch and yaw) during the land vehicle movement and performing a virtual rotation of the video camera, while the method has the advantage of reducing the required computing power during image processing.
In the claimed method, virtual rotation of the video camera is understood as such a transformation of an image to a new image, which could be obtained from a video camera rotated at a given angle.
The technical result of the claimed invention consists in improving the accuracy of calibration of external parameters of video cameras.
The technical result is achieved by implementing a method for calibrating the external parameters of video cameras, characterized by the fact that during the movement of a vehicle, images of the environment are obtained from video cameras, linear features of the environment are identified in the image, an image with linear features and rectilinear segments is formed, the position of the vanishing point for the determined rectilinear segments is identified as the position of the intersection point of the formed rectilinear segments by determining the parameters of the function describing the straight line formed from sets of points characterizing the totality of these rectilinear segments, the vanishing point position is compared with the principal point position; based on the deviation obtained, the values of the camera's rotation angles are determined, taking into account the focal lengths known from the internal calibration of the camera, and an adjusted image is formed, taking into determined camera rotation angles.
In the preferred embodiment of the method for calibrating the external parameters of video cameras, rectilinear segments in the image are formed by using a fast Hough transform.
As part of the presentation of the proposed method, the following explanations, clarifying the meaning of the terms used, are introduced.
Thus, the vanishing point is understood as the point of intersection of straight lines parallel in the real world.
The principal point is the point of intersection of the image plane with 10the optical axis of a video camera.
The method for calibrating of video cameras is preferably implemented as follows (
During the LV movement, the front video cameras register images of linear environmental features of the carriageway along which this vehicle is moving (1).
It should be noted that in order to implement the method, the LV moves parallel to rectilinear features located in the video camera coverage area (road markings, edges of buildings and roads).
Images from video cameras are sent to the onboard computer and processed independently of each other. Based on this, all subsequent stages are considered from the point of view of a single video camera. The original image is preprocessed, which implies correcting radial distortion to eliminate distortions of the observed objects caused by the lens of the video camera, and then the image is converted to grayscale (2), then linear features are identified in the image, preferably by the Canny algorithm, after which the found linear features are combined into one image by pixel addition (3).
At the next stage, the image is smoothed with a Gaussian filter to compensate for errors in detecting linear environmental features (4). After that, rectilinear segments intersecting at the vanishing point are highlighted in the smoothed image.
Rectilinear segments in the image are formed using the application of the fast Hough transform (FHT), according to which a Hough image is constructed (5), then the search area for the vanishing point on the Hough image is determined.
To identify the area of the vanishing point search, a black-and-white image is formed, where the neighborhood of the expected point is marked in white, FHT is applied to the resulting image, the coordinates of the leftmost and rightmost pixels with a non-zero value are determined on the resulting Hough image, while the values of these pixels on the X-axis set the search range for the vanishing point.
To highlight rectilinear segments on the Hough image, the brightest white pixel in the image is registered as the maximum, which is initially represented as a y=kx+b straight line, after that its neighborhood is removed from the Hough image by assigning zero values to the corresponding pixels.
The determined maximum point with coordinates (k, b) is transformed back into a rectilinear segment, described by a y=kx+b function, when it is superimposed on the original image.
While forming a set of rectilinear segments, the found segment is checked to determine if it is horizontally or vertically oriented, and in case of an angular mismatch of this segment from the vertical or horizontal straight line by less than a given threshold angular value, this segment is excluded from the set of rectilinear segments. This step allows to reduce the negative impact of additional objects in the camera coverage (roofs of cars, trees, etc.).
After forming a set of rectilinear segments in the original image, the position of the vanishing point (7) is determined as the intersection point of the rectilinear segments highlighted in the image. To this end, the set of selected rectilinear segments, each of which is described by a y=kx+b function, is transformed into a parameter space (k, b) in which each rectilinear segment is represented as a point with coordinates (k, b).
With such parameterization, the set of these rectilinear segments intersecting in the original image is represented by a set of points lying on one straight line, characterized by the parameters {circumflex over (k)}, {circumflex over (b)} and described by a y={circumflex over (k)}x+{circumflex over (b)} function.
To determine these parameters, lines are constructed between all possible pairs of points, and then, the average value is determined among the set of slope coefficients of these lines, which is the slope coefficient of the line {circumflex over (k)} characterizing the position of the vanishing point. The intercept {circumflex over (b)} is determined by the found coefficient {circumflex over (k)} as the average value among the set of intercepts of the constructed lines with a constant slope coefficient {circumflex over (k)}.
The resulting straight line y={circumflex over (k)}x+{circumflex over (b)} is transformed into a point on the original image with coordinates ({circumflex over (k)}, {circumflex over (b)}) by applying it to a rectangular coordinate system.
Thus, the constructed point on the original image with coordinates ({circumflex over (k)}, {circumflex over (b)}) is the vanishing point.
In order to evaluate the external calibration parameters of video cameras, the vanishing point position is compared with the principal point position, and the camera rotation angles are determined based on the deviation obtained, taking into account the focal lengths known from the internal calibration of the camera as follows:
According to the assessment of change in camera rotation angles during the vehicle movement, their values are corrected and virtual rotation of the video camera is carried out.
Thus, the proposed method for calibrating the external parameters of video cameras with high accuracy and computational efficiency makes it possible to estimate the pitch and yaw angles of the video camera during the LV movement and can be widely used as part of autonomous LV localization systems in order to increase the accuracy of determining its own position.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021123500 | Aug 2021 | RU | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/RU2022/050200 | 6/26/2022 | WO |
