Patent Application
20220254065
This application claims priority to Chinese Patent Application No. 202110175040X, filed on Feb. 9, 2021, which is hereby incorporated by reference in its entirety.
The present application relates to image processing technologies, and in particular, to a camera calibration method, apparatus and an electronic device.
Parameters of a geometric model of camera imaging are called camera parameters. The camera parameters include camera internal parameters such as focal length and imaging center, and camera external parameters such as rotation matrix and translation matrix. The process of determining the camera internal and external parameters is called camera calibration.
In applications such as image measurement or machine vision, camera calibration is a key part, and the accuracy of a camera calibration result directly affects the accuracy of a result produced by camera operation. When camera calibration is performed, a picture of a calibration object with a solid circle pattern is taken, and image coordinates of the center of the solid circle are further determined, and internal and external parameters of a camera are determined according to the image coordinates of the center of the circle and known world coordinates of the center of the circle. The calibration accuracy is low by adopting this solution.
The present application provides a camera calibration method, apparatus and an electronic device, by which the accuracy of camera calibration is improved.
In a first aspect, the present application provides a camera calibration method, including:
obtaining a calibration board image, where the calibration board image includes a plurality of annular patterns;
obtaining an inner edge and an outer edge of each annular pattern in the calibration board image;
determining image coordinates of a center point of each annular pattern according to the inner edge and the outer edge of each annular pattern; and
determining internal and external parameters of a camera according to the image coordinates and corresponding world coordinates of the center point of each annular pattern.
In a feasible implementation, the obtaining the inner edge and the outer edge of each annular pattern in the calibration board image includes:
determining an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image, and determining the inner edge and the outer edge of each annular pattern in the pixel level as the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the obtaining the inner edge and the outer edge of each annular pattern in the calibration board image includes:
determining an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image; and
performing smoothing processing on the inner edge and the outer edge in the pixel level to obtain an inner edge and an outer edge in a sub-pixel level, and determining the inner edge and the outer edge in the sub-pixel level as the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the determining the image coordinates of the center point of each annular pattern according to the inner edge and the outer edge of each annular pattern includes:
dividing an area between the inner edge and the outer edge of each annular pattern to obtain a plurality of divided blocks;
calculating gravity center of each divided block; and
fitting the gravity centers of the plurality of divided blocks, and determining the image coordinates of the center point of each annular pattern according to a fitting result.
In a feasible implementation, the dividing the area between the inner edge and the outer edge of each annular pattern includes:
dividing the area between the inner edge and the outer edge of each annular pattern according to a normal of the inner edge and/or the outer edge of each annular pattern.
In a feasible implementation, the method further includes:
determining a relative positional relationship between the image coordinates of the center point of each annular pattern in the calibration board image;
and determining a corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern according to the relative positional relationship and the predetermined world coordinates of the center point of each annular pattern in the calibration board image.
In a feasible implementation, the determining the internal and external parameters of the camera according to the image coordinates and the corresponding world coordinates of the center point of each annular pattern includes: determining the internal and external parameters of the camera according to the corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern and a mapping relationship between an image coordinate system and a world coordinate system.
In a second aspect, the present application provides a camera calibration apparatus, including:
a first obtaining module, configured to obtain a calibration board image, where the calibration board image includes a plurality of annular patterns;
a second obtaining module, configured to obtain an inner edge and an outer edge of each annular pattern in the calibration board image;
a determining module, configured to determine image coordinates of a center point of each annular pattern according to the inner edge and the outer edge of each annular pattern; and a calibrating module, configured to determine internal and external parameters of a camera according to the image coordinates and corresponding world coordinates of the center point of each annular pattern.
In a feasible implementation, the second obtaining module is configured to:
determine an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image, and determine the inner edge and the outer edge of each annular pattern in the pixel level as the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the second obtaining module is configured to:
determine an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image; and
perform smoothing processing on the inner edge and the outer edge in the pixel level to obtain an inner edge and an outer edge in a sub-pixel level, and determine the inner edge and the outer edge in the sub-pixel level as the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the determining module is configured to:
divide an area between the inner edge and the outer edge of each annular pattern to obtain a plurality of divided blocks;
calculate gravity center of each divided block; and
fit the gravity centers of the plurality of divided block, and determine the image coordinates of the center point of each annular pattern according to a fitting result.
In a feasible implementation, the determining module is configured to:
divide the area between the inner edge and the outer edge of each annular pattern according to a normal of the inner edge and/or the outer edge of each annular pattern.
In a feasible implementation, the calibrating module is configured to:
determine a relative positional relationship between the image coordinates of the center point of each annular pattern in the calibration board image; and
determine a corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern according to the relative positional relationship and the predetermined world coordinates of the center point of each annular pattern in the calibration board image.
In a feasible implementation, the calibrating module is configured to:
determine the internal and external parameters of the camera according to the corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern and a mapping relationship between an image coordinate system and a world coordinate system.
In a third aspect, the present application provides an electronic device, including: a memory and a processer, and the memory is connected with the processer;
where the memory is configured to store a computer program;
the processer is configured to implement the method according to the first aspect when the computer program is executed.
In a forth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, where when the computer program is executed by a processor, the method according to the first aspect is implemented.
In a fifth aspect, the present application provides a computer program product, including a computer program, where when the computer program is executed by a processor, the method according to the first aspect is implemented.
The present application provides a camera calibration method, apparatus and an electronic device. The pattern of the calibration board is improved, and the calibration board with the annular pattern is used for camera calibration. Because the annular pattern has two edges of the inner circle and the outer circle, the image coordinates of the center point of the annular pattern are jointly determined by these two edges. Compared with the image coordinates of the center determined by the solid circle method, the image coordinates of the center point obtained by using the two edges of the annular pattern in this embodiment are more accurate, so that the calibration result of the camera is more accurate.
In order to explain technical solutions of the embodiments of the present application or the prior art more clearly, drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced in the following. Obviously, the drawings in the following description are some embodiments of the present application, and for those of ordinary skilled in the art, other drawings can be obtained according to these drawings without paying creative effort.
In order to make the purpose, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skilled in the art without paying creative effort shall fall within the scope of protection of the present application.
In various applications such as image measurement, machine vision and etc., it is often necessary to calculate geometric information of an object in a three-dimensional space from image information obtained by a camera, for example, reconstructing a three-dimensional object, identify depth information of an object, etc. a relationship between a three-dimensional position of a point on surface of an object and a corresponding point of the object in an image is determined by a geometric model imaged by the camera, that is, camera parameters. Therefore, calibration of the camera parameters directly affects the accuracy of a result produced by the camera.
Camera calibration can be realized by means of a three-dimensional or two-dimensional calibration board, the calibration board is used to provide a point with known coordinate information (world coordinates), and also known as a target point. The camera is used to shoot the calibration board, and image coordinates of the target point are extracted by processing the obtained image of the calibration board, and internal and external parameters of the camera are determined based on the image coordinates and world coordinates of each target point. The world coordinate is a world coordinate system based on three-dimensional world defined by users, which is used to describe a position of the calibration board in the three-dimensional world. The image coordinates are image pixel coordinates, which are used to describe positions of pixels in the image. World coordinates and image coordinates are expressed in Cartesian coordinates.
Because of the complexity of production of the three-dimensional calibration board, the two-dimensional calibration board is usually used in practical applications. The calibration board has specific patterns, for example, the calibration board has solid circles arranged at intervals. After the calibration board is placed, the world coordinates of the center of the solid circle on the calibration board (that is, the target point) can be determined. Then, the calibration board is shot by using the camera, the obtained image is processed, the image coordinates of the center of the solid circle in the image are extracted, and then the internal and external parameters of the camera are determined according to the image coordinates and world coordinates of the center of the solid circle.
In the above camera calibration process, because the world coordinates of the target point can be obtained in advance and the world coordinates are relatively accurate, the accuracy of the camera calibration result is mainly determined by the accuracy of the image coordinates of the target point, that is, the more accurate the image coordinates of the target point extracted from the image, the more accurate the camera calibration result is.
However, the accuracy of extracting the image coordinates of the center of the solid circle from the image is usually low when the calibration board with the solid circle pattern is used for the camera calibration. Especially for a camera with low resolution, such as a time of flight (Time of flight, TOF) depth camera, the accuracy of extracting the image coordinates of the center of a solid circle from the image will be lower, which will lead to a more inaccurate calibration result.
In order to improve the accuracy of the camera calibration, the present application provides a camera calibration method, which improves the pattern of the calibration board, and proposes to use the calibration board with an annular pattern to calibrate a camera. Because the annular pattern has two edges from inner circle and outer circle, the inner circle and outer circle are used to jointly determine a center point of the annular pattern, and the obtained image coordinates of the center point are more accurate, and thus a calibration result of the camera is more accurate.
In the following, the camera calibration method provided by the present application will be described in detail through specific examples. It can be understood that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
S101: obtain a calibration board image.
The calibration board image includes a plurality of annular patterns. Exemplarily, the calibration board adopted in the embodiment of the present application has an annular pattern as show in
In order to achieve the camera calibration, the calibration board image needs to include at least three perspectives of the calibration board. There are two ways for the camera to shoot the calibration board. In one way, a calibration board may be shot several times, and the calibration board includes more than four annular patterns. A pose of the calibration board or the camera is adjusted per shooting. In this way, one calibration board may occupy whole field of view of the camera as much as possible, and the number of target points in the calibration board may be more, which is beneficial to improve the calibration accuracy.
In the other way, a plurality of calibration boards may be shot once, and each calibration board includes more than four annular patterns. Exemplarily, as shown in
S102: obtain an inner edge and an outer edge of each annular pattern in the calibration board image.
S103: determine image coordinates of a center point of each annular pattern according to the inner edge and the outer edge of each annular pattern.
Because the color of the annular pattern in the calibration board is obviously different from the color of the background part of the calibration board, the edge of each annular pattern may be obtained by performing edge detection on the calibration board image, and the edge of the annular pattern includes the inner edge and the outer edge, that is, an inner circle and an outer circle.
A circular annular range of the annular pattern is jointly determined by the inner edge and the outer edge of the annular pattern, so that the image coordinates of the center point of the annular pattern may be jointly determined according to the inner edge and the outer edge.
S104: determine internal and external parameters of a camera according to the image coordinates and corresponding world coordinates of the center point of each annular pattern.
The world coordinates of the center point of each annular pattern can be converted into the image coordinates of the center point of each annular pattern by transformation of a matrix composed of the internal and external parameters of the camera. In this embodiment, since the calibration board is placed in advance according to a certain pose, the world coordinates of the center point of each annular pattern in the calibration board may be determined in advance. Therefore, after the image coordinates of each center point are extracted from the image of the calibration board, the internal and external parameters of the camera can be determined according to the image coordinates and the corresponding world coordinates to complete the calibration of the camera.
According to the camera calibration method provided by this embodiment, the pattern of the calibration board is improved, and the calibration board with the annular pattern is used for camera calibration. Because the annular pattern has two edges from the inner circle and the outer circle, the image coordinates of the center point of the annular pattern are jointly determined by these two edges. Compared with the image coordinates of the center point determined by the solid circle method, the image coordinates of the center obtained by using the two edges of the annular pattern in this embodiment are more accurate, so that the calibration result of the camera is more accurate.
Through the above calibration of the camera, accurate camera parameters can be obtained, which can further make the result produced by the camera operation more accurate. Exemplarily, for a TOF depth camera, after the camera is calibrated, depth information of the subject determined based on the camera parameters will be more accurate.
On the basis of the above embodiment, the image coordinates of the center point of each annular pattern extracted from the calibration board image will be further explained.
S401: obtain a calibration board image.
S401 in this embodiment is similar to S101 in the embodiment shown in
S402: determine an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image.
Because the color of the annular pattern in the calibration board is obviously different from the color of the background part of the calibration board, the inner edge and outer edge of the annular pattern can be determined by detecting a gray value of each pixel in the calibration board image and according to gray value change. Exemplarily, through setting a gray change threshold, when the gray scale value change of adjacent pixels is greater than the gray change threshold, the pixel with a lower gray scale value is determined as the pixel at the edge of the annular pattern. Through detecting the gray values of the pixels in the whole calibration board image, the inner edge and outer edge of each annular pattern in the pixel level can be determined.
S403: perform smoothing processing on the inner edge and the outer edge in the pixel level to obtain an inner edge and an outer edge in a sub-pixel level.
For a camera with a lower resolution, that is, a camera with a lower pixel, performing subsequent processing based on the edge in the pixel level may result in a low accuracy. Therefore, in this embodiment, the inner edge and the outer edge in the pixel level are further smoothed. Exemplarily, an interpolation method is used to interpolate the gray values of pixels at the inner edge and outer edge in the pixel level to obtain the inner edge and the outer edge in the sub-pixel level. Exemplarily, the inner edge and the outer edge in the sub-pixel level may also be obtained by fitting the gray values of pixels of the inner edge and the outer edge in the pixel level.
It should be noted that in S403, in order to improve the calibration accuracy, the inner and outer edges of the annular pattern are smoothed to obtain the inner edge and the outer edge in the sub-pixel level. In the actual calibration process, S403 may be executed or skipped as needed. That is, in an implementation, only S401-S402 may be performed, and after determining the inner and outer edges of each annular pattern in the pixel level, the inner and outer edges of each annular pattern in the pixel level will be determined as the inner and outer edges of each annular pattern. In another implementation, S401-S403 are performed, that is, the inner and outer edges of each annular pattern in the sub-pixel level are determined, and the inner and outer edges of each annular pattern in the sub-pixel level are determined as the inner and outer edges of each annular pattern. Thereafter, the subsequent S404-S408 is further performed based on the determined inner edge and outer edge of each annular pattern.
S404: divide an area between the inner edge and the outer edge of each annular pattern to obtain a plurality of divided blocks, and calculate gravity center of each divided block.
After the inner edge and outer edge of each annular pattern are determined, the annular area between the inner edge and the outer edge is divided, in an implementation, normals at a plurality of positions of the inner edge and/or the outer edge are determined, and the annular area between the inner edge and the outer edge of each annular pattern is divided according to the normals of the inner edge and/or the outer edge of each annular pattern. Taking the normal of the inner edge as an example, the inner edge is divided into a plurality of arc segments. Arc AB shown in
For each divided block obtained by division, the image coordinates of its gravity center position are calculated. Exemplarily, the gravity center of the divided block may be determined according to the gray value of the divided block through the gray gravity center method. The three points shown in
S405: fit the gravity centers of the plurality of divided blocks, and determine image coordinates of the center point of each annular pattern according to a fitting result.
After the gravity centers of the plurality of divided blocks are determined, the gravity centers of the plurality of divided blocks are fitted. Exemplarily, the gravity centers of the plurality of divided blocks may be meshed by means of average method, least square method, Gaussian fitting and so on, so as to determine the image coordinates of the center point of the annular pattern.
S406: determine a relative positional relationship between the image coordinates of the center point of each annular pattern in the calibration board image.
Since the camera calibration depends on conversion between the image coordinates and the world coordinates of the target point (the center point of the annular pattern in this embodiment), after the image coordinates of the center point of each annular pattern are extracted, it is necessary to determine a relative positional relationship between the center point of each annular pattern according to values in a horizontal axis and a vertical axis of the coordinates, so as to determine a corresponding relationship between the image coordinates and the world coordinates.
S407: determine a corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern according to the relative positional relationship and predetermined world coordinates of the center point of each annular pattern in the calibration board image.
The relative positional relationship between the image coordinates the center point of each annular pattern may be determined according to the image coordinates of the center point of each annular pattern, and then a one-to-one corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern may be determined through further combining the predetermined world coordinates of the center point of each annular pattern in the calibration board image.
S408: determine internal and external parameters of a camera according to the corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern and a mapping relationship between an image coordinate system and a world coordinate system.
The mapping relationship between the image coordinate system and the world coordinate system consists of internal and external parameter matrices of the camera. When parameter calibration is performed, firstly, a homography matrix corresponding to each calibration board is determined according to the image coordinates and the world coordinates of the center point of the annular pattern in each calibration board, and the internal parameters of the camera are determined by homography matrices of a plurality of calibration boards. Further, the external parameters of the camera are determined by the homography matrix and the internal parameters.
According to the camera calibration method provided by this embodiment, the camera calibration is performed by using the calibration board with the annular pattern. Because the annular pattern has two edges from the inner circle and the outer circle, the image coordinates of the center point of the annular pattern are jointly determined by these two edges. When edge detection is performed, the inner edge and the outer edge in the pixel level may be optimized to obtain the inner edge and the outer edge in the sub-pixel level, thereby the edge accuracy is further improved. It is more accurate to determine the image coordinates of the center point of the annular pattern according to the two high-precision annular edges than to determine the image coordinates of the center of the circle by using the solid circle method, which makes the calibration result of the camera more accurate.
a first obtaining module 601, configured to obtain a calibration board image, where the calibration board image includes a plurality of annular patterns;
a second obtaining module 602, configured to obtain an inner edge and an outer edge of each annular pattern in the calibration board image;
a determining module 603, configured to determine image coordinates of a center point of each annular pattern according to the inner edge and the outer edge of each annular pattern; and a calibrating module 604, configured to determine internal and external parameters of a camera according to the image coordinates and corresponding world coordinates of the center point of each annular pattern.
In a feasible implementation, the second obtaining module 602 is configured to:
determine an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image, and determine the inner edge and the outer edge of each annular pattern in the pixel level as the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the second obtaining module 602 is configured to:
determine an inner edge and an outer edge of each annular pattern in a pixel level according to gray values of pixels in the calibration board image; and
perform smoothing processing on the inner edge and the outer edge in the pixel level to obtain an inner edge and an outer edge in a sub-pixel level, and determine the inner edge and the outer edge in the sub-pixel level as the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the determining module 603 is configured to:
divide an area between the inner edge and the outer edge of each annular pattern to obtain a plurality of divided blocks;
calculate gravity center of each divided block;
fit the gravity centers of the plurality of divided block, and determine the image coordinates of the center point of each annular pattern according to a fitting result.
In a feasible implementation, the determining module 603 is configured to:
divide the area between the inner edge and the outer edge of each annular pattern according to a normal of the inner edge and the outer edge of each annular pattern.
In a feasible implementation, the calibrating module 604 is configured to:
determine a relative positional relationship between the image coordinates of the center point of each annular pattern in the calibration board image; and
determine a corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern according to the relative positional relationship and the predetermined world coordinates of the center point of each annular pattern in the calibration board image.
In a feasible implementation, the calibrating module 604 is configured to:
determine the internal and external parameters of the camera according to the corresponding relationship between the image coordinates and the world coordinates of the center point of each annular pattern and a mapping relationship between an image coordinate system and a world coordinate system.
The camera calibration apparatus provided by this embodiment may be configured to execute the camera calibration method in the above method embodiment, and their implementation principle and calculation effect are similar, which are not repeated here.
The memory 701 is configured to store a computer program.
The processer 702 is configured to implement the camera calibration method in the above method embodiments when the computer program is executed.
In an implementation, the electronic device may be a computer device, a server, etc. for processing an image taken by a camera and calculating internal and external parameters of the camera. Alternatively, the electronic device may also be an electronic device with a camera, and the electronic device may calibrate the parameters of its own camera.
The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, where when the computer program is executed by a processor, the camera calibration method in the above method embodiment is implemented.
The embodiment of the present application also provides a computer program product, including a computer program, where when the computer program is executed by a processor, the camera calibration method in the above method embodiment is implemented.
It can be understood by those of ordinary skilled in the art that all or part of the steps to implement the above-mentioned method embodiments may be accomplished by hardware related to the program instructions. The aforementioned programs may be stored in a computer-readable storage medium. When the program is executed, the steps including the above-mentioned method embodiments are executed, the aforementioned storage medium includes a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and other media that can store program codes.
Finally, it should be illustrated that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skilled in the art should understand that it is still possible to modify the technical solutions described in the foregoing embodiments or to replace some or all of the technical features equivalently thereof. However, these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solution of each embodiment of the present application.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202110175040.X | Feb 2021 | CN | national |
