Patent Application
20220036587
The present disclosure relates to the technical field of computer vision, and more particularly, to a calibration method, an electronic device, and a storage medium.
With rapid development of artificial intelligence technologies, traditional industries and information technologies work together to bring convenience to people's lives. For example, with the combination of automotive industries and information technologies, intelligence vehicles capable of automatic driving have been produced. During automatic driving of intelligence vehicles, ranging is important. Among various ranging sensors for intelligence automobile driver-assistance, visual sensors can obtain richer information on road structure and environment and are relatively cheap.
Among vision ranging technologies, monocular-vision ranging technologies are widely used due to their advantages of low cost, simple system installation, and good stability etc. compared with multi-vision ranging technology. In monocular-vision ranging, a homography matrix is adopted. Based on a pixel coordinate of a target object in an image coordinate system and the homography matrix, a world coordinate of the target object in a world coordinate system can be obtained. Based on the world coordinate, information on a distance between the target object and a preset location can be obtained. Therefore, the accuracy of the homography matrix directly affects the accuracy of a ranging result.
The homography matrix is obtained by calibration in advance. During calibration, a world coordinate of a reference object in the world coordinate system is known, and the reference object needs to be selected from an image containing the reference object taken by an image acquisition device, to obtain a pixel coordinate of the reference object in the image coordinate system. Generally, the reference object needs to be manually selected from the image. Due to visual errors, the selection is inaccurate, which leads to an inaccurate calibration result.
According to a first aspect, a calibration method is provided. The calibration method includes the following. A sample image is acquired with an image acquisition device. Initial pixel coordinates of multiple sample reference objects in the sample image in an image coordinate system are determined based on the sample image. Straight-line fitting is performed on sample reference objects on a same straight line based on the determined initial pixel coordinates of the sample reference objects in the image coordinate system, and the initial pixel coordinates of the sample reference objects involved in straight-line fitting are corrected based on fitted straight lines to obtain corrected pixel coordinates. A homography matrix of the image acquisition device is determined based on world coordinates of the multiple sample reference objects in the sample image in a world coordinate system and the obtained corrected pixel coordinates.
According to a second aspect, an electronic device is provided. The electronic device includes a processor, a bus, and a storage. The storage is configured to store machine-readable instructions to be executed by the processor, where when the electronic device is running, the processor communicates with the storage through the bus, and the processor executes the machine-readable instructions to: acquire a sample image with an image acquisition device; determine initial pixel coordinates of multiple sample reference objects in the sample image in an image coordinate system based on the sample image; perform straight-line fitting on sample reference objects on a same straight line based on the determined initial pixel coordinates of the sample reference objects in the image coordinate system, and correct the initial pixel coordinates of the sample reference objects involved in straight-line fitting based on fitted straight lines to obtain corrected pixel coordinates; determine a homography matrix of the image acquisition device based on world coordinates of the multiple sample reference objects in the sample image in a world coordinate system and the obtained corrected pixel coordinates.
According to a third aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to: acquire a sample image with an image acquisition device; determine initial pixel coordinates of multiple sample reference objects in the sample image in an image coordinate system based on the sample image; perform straight-line fitting on sample reference objects on a same straight line based on the determined initial pixel coordinates of the sample reference objects in the image coordinate system, and correct the initial pixel coordinates of the sample reference objects involved in straight-line fitting based on fitted straight lines to obtain corrected pixel coordinates; determine a homography matrix of the image acquisition device based on world coordinates of the multiple sample reference objects in the sample image in a world coordinate system and the obtained corrected pixel coordinates.
For describing the technical solutions of implementations of the present disclosure more clearly, the drawings required to be used in implementations will be simply introduced below. The drawings, which are incorporated in and constitute a part of this specification, illustrate implementations consistent with the present disclosure and, together with the specification, serve to explain the technical solutions of the present disclosure. It is to be understood that the following drawings only illustrate some implementations of the present disclosure and thus should not be considered as limits to the scope. Those of ordinary skill in the art may also obtain other related drawings according to these drawings without creative work.
To make the purposes, technical solutions, and advantages of implementations of the present disclosure clearer, the technical solutions in the implementation will be clearly and completely described below in combination with the drawings in the implementation. It is apparent that the described implementations are not all implementations but only part of implementations of the present disclosure. Components, described and shown in the drawings, in the implementations of the present disclosure may usually be arranged and designed with various configurations. Therefore, the following detailed description for the implementations of the present disclosure in the drawings is not intended to limit the claimed scope of the present disclosure but only represents selected implementations of the present disclosure. Other implementations obtained by those of ordinary skill in the art based on the implementations of the present disclosure without creative work shall fall within the scope of the present disclosure.
At present, in the field of automatic driving and robotics, vision ranging is often carried out with an image acquisition device. The principle of vision ranging with the image acquisition device is as follows. A pixel coordinate of a target object captured by the image acquisition device in an image coordinate system is determined. A world coordinate of the target object in a world coordinate system is determined based on a homography matrix of the image acquisition device. A distance between a preset location and the target object is determined according to a world coordinate of the preset location and the world coordinate of the target object. The preset location can be the origin of the world coordinate system. Therefore, the accuracy of the homography matrix directly affects the accuracy of a ranging result. The homography matrix is obtained by calibration in advance. During calibration, a world coordinate of a reference object in the world coordinate system is known, and the reference object needs to be picked up from an image containing the reference object taken by an image acquisition device, to obtain a pixel coordinate of the reference object in the image coordinate system. Generally, the reference object needs to be manually selected from the image. Due to visual errors, the selection may be inaccurate, which leads to an inaccurate calibration result. In view of this, a method for correcting a pixel coordinate of a selected reference object is provided according to implementations of the present disclosure.
Based on the foregoing, a calibration method is provided. After a sample image containing sample reference objects is acquired with an image acquisition device, initial pixel coordinates of the sample reference objects in an image coordinate system are determined. Straight-line fitting is performed on sample reference objects in the sample image on a same straight line based on the initial pixel coordinates of the sample reference objects in the image coordinate system. The initial pixel coordinates of the sample reference objects involved in straight-line fitting are corrected based on fitted straight lines to obtain corrected pixel coordinates of the sample reference objects involved in straight-line fitting in the image coordinate system.
The sample reference objects can be placed in advance. For example, the sample reference objects can be arranged in a sample reference object array, so that sample reference objects in a same row or column are on a same straight line in the world coordinate system. The initial pixel coordinates of the sample reference objects in the image coordinate system can be corrected by performing straight-line fitting with the initial pixel coordinates of the sample reference objects in the image coordinate system, to obtain the corrected pixel coordinates with high accuracy of the sample reference objects in the image coordinate system. Then, according to the world coordinates of the sample reference objects in the sample reference object array in the world coordinate system and the corrected pixel coordinates of the sample reference objects in the image coordinate system, an accurate homography matrix of the image acquisition device can be obtained. In this way, the accuracy of calibration for the image acquisition device can be improved.
It should be noted that in the drawings, similar reference numbers indicate similar items. Therefore, once a certain item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.
For better understanding of implementations, a calibration method according to implementations of the present disclosure is first illustrated in detail. Generally, the calibration method provided is executed by a computer device with data processing capabilities.
At S101, a sample image is acquired with an image acquisition device.
At S102, initial pixel coordinates of multiple sample reference objects in the sample image in an image coordinate system are determined based on the sample image.
At S103, straight-line fitting is performed on sample reference objects on a same straight line based on the determined initial pixel coordinates of the sample reference objects in the image coordinate system, and the initial pixel coordinates of the sample reference objects involved in straight-line fitting are corrected based on fitted straight lines to obtain corrected pixel coordinates.
At S104, a homography matrix of the image acquisition device is determined based on world coordinates of the multiple sample reference objects in the sample image in a world coordinate system and the obtained corrected pixel coordinates.
The above S101-S104 are respectively described below.
At S101, the sample image can be acquired with the image acquisition device as follows. The sample image can be acquired by shooting a sample reference object array with the image acquisition device. An image acquisition environment and a world coordinate system where sample reference objects are located can be set in advance. For example, the image acquisition environment can be set by drawing multiple straight lines on the ground or looking for a place containing multiple lane lines to form multiple straight lines L, and placing the sample reference objects of a same shape on each straight line L. For example, the sample reference objects can be cone sample reference objects. The sample reference objects can be divided into multiple groups, and each group of sample reference objects are located on a same straight line L. In addition, multiple straight lines H need to be provided so that each straight line H intersects each straight line L. The sample reference objects are placed at intersections of the straight lines L and the straight lines H to obtain the sample reference object array. The sample reference object array includes the sample reference objects collinear on both the multiple straight lines L and the multiple straight lines H.
In some examples, a world coordinate system is established with a mapping point of the center of a front axle of a vehicle or the center of the body of the vehicle on the ground as the origin. The origin is a set location point, and the image acquisition device is located at a set location of the vehicle. In this way, a world coordinate system as shown in
In some examples, a camera of the image acquisition device on the vehicle is adjusted to be parallel to the ground. When the image acquisition device shoots the sample reference object array in a Y-axis direction, a sample image as shown in
At S102, after the sample image of the sample reference objects is obtained, the sample image can be placed in the image coordinate system. Based on the location where a cone sample reference object, which is manually selected by a user in the image coordinate system, is tangent to the ground, the initial pixel coordinate of the cone sample reference object in the image coordinate system can be determined. Alternatively, the sample image can be input into a pixel coordinate determination model trained in advance to determine the initial pixel coordinate of each sample reference object.
The pixel coordinate determination model can be configured to perform image recognition based on the sample image to determine, for each cone sample reference object, the location where the cone sample reference object is tangent to the ground, the location thus determined is the location of the cone sample reference object in the image; then determine the initial pixel coordinate of the cone sample reference object in the image coordinate system based on the location of the cone sample reference object in the image.
At S103, in the world coordinate system, sample reference objects on each straight line L are collinear, sample reference objects on each straight line H are collinear, and each sample reference object is located at an intersection of a straight line L and a straight line H. In view of this, straight-line fitting can be performed on the sample reference object array in the sample image to correct the initial pixel coordinate of each sample reference object.
As shown in
At S401, straight-line fitting is performed on the sample reference objects on each straight line in a first direction respectively based on the determined initial pixel coordinates of the sample reference objects in the image coordinate system, to obtain multiple first fitted straight lines.
Line segments between sample reference objects on one first fitted straight line do not intersect line segments between sample reference objects on another first fitted straight line. For example, the multiple first fitted straight lines are parallel to each other, or the multiple first fitted straight lines intersect in the distance but do not intersect at a position where the sample reference objects are located.
The determined initial pixel coordinates may be inaccurate due to visual errors or errors of the pixel coordinate determination model. As a result, the initial pixel coordinates that should be on a same straight line may not be on the same straight line. In view of this, straight-line fitting can be performed with the initial pixel coordinates to obtain the multiple first fitted straight lines.
After straight-line fitting is performed on the sample reference objects on each straight line in the first direction respectively, line segments between sample reference objects on one first fitted straight line do not intersect line segments between sample reference objects on another first fitted straight line. For example, after straight-line fitting is performed on the sample reference objects, multiple first fitted straight lines corresponding to straight lines L in
For example, first straight-line fitting can be performed on sample reference objects on straight lines in the first direction as follows. The sample reference objects in the sample image can be divided into multiple groups of sample reference objects. Each group of sample reference objects belong to a same straight line in the world coordinate system. In one case, the sample reference objects can be grouped according to which straight line L they belong to in the world coordinate system. That is, sample reference objects belonging to a same straight line L in the world coordinate system are divided into one group. In another case, the sample reference objects can be grouped according to which straight line H they belong to in the world coordinate system. That is, sample reference objects belonging to a same straight line H in the world coordinate system are divided into one group.
Taking the latter case as an example, the sample reference objects belonging to the same straight line L in the world coordinate system are divided into one group. For example, in the sample image shown in
For each group of sample reference objects, first straight-line fitting can be performed with the initial pixel coordinates of the group of sample reference objects by adopting the least square method. A first straight-line equation corresponding to a first fitted straight line can be obtained according to the following equations (1) -(3):
where (xi, yi) indicates an initial pixel coordinate of the i-th sample reference object in the group of sample reference objects, n indicates the number of sample reference objects in the group,
For each group of sample references objects, after the initial pixel coordinates corresponding to the group of sample reference objects are substituted into equations (1)-(3), the unknown parameters b0 and b1 in the first straight-line equation corresponding to the group of sample reference objects can be obtained. In this way, the first straight-line equation, y=b1x−b0, of the first fitted straight line corresponding to the group of sample reference objects can be obtained.
At S402, the initial pixel coordinates of the sample reference objects in the image coordinate system are corrected based on the multiple first fitted straight lines to obtain intermediate pixel coordinates, and straight-line fitting is performed on the sample reference objects on each straight line in a second direction respectively based on the intermediate pixel coordinates of the sample reference objects to obtain multiple second fitted straight lines.
Each straight line in the first direction intersects each straight line in the second direction.
Each initial pixel coordinate includes an initial first coordinate value and an initial second coordinate value. A first coordinate axis corresponding to the initial first coordinate value is perpendicular to a second coordinate axis corresponding to the initial second coordinate value.
The initial pixel coordinates of the sample reference objects in the image coordinate system can be corrected based on the multiple first fitted straight lines to obtain the intermediate pixel coordinates as follows. For each sample reference object, the initial first coordinate value of the initial pixel coordinate is substituted into a straight-line equation of the first fitted straight line where the sample reference object is located to obtain an intermediate second coordinate value. The intermediate pixel coordinate of the sample reference object includes the initial first coordinate value and the intermediate second coordinate value of the sample reference object. The process of correcting the initial pixel coordinates to obtain the intermediate pixel coordinates can be regarded as the process of correcting the initial second coordinate values of the sample reference objects.
In the image coordinate system, the first coordinate axis may be a horizontal axis or a vertical axis. When the first coordinate axis is the horizontal axis of the image coordinate system, the second coordinate axis is the vertical axis of the image coordinate system. Alternatively, when the first coordinate axis is the vertical axis of the image coordinate system, the second coordinate axis is the horizontal axis of the image coordinate system.
For example, the sample image in
The initial first coordinate value substituted into the straight-line equation of the first fitted straight line can be the initial horizontal coordinate value corresponding to the horizontal axis or the initial vertical coordinate value corresponding to the ordinate axis. In many historical experiments, it has been found that no matter whether the initial pixel coordinate is obtained by manually marking the sample reference object in the sample image or determined with the pixel coordinate determination model, in a determined initial pixel coordinate, the accuracy of an initial horizontal coordinate value is higher than the accuracy of an initial vertical coordinate value. Therefore, in implementations of the present disclosure, initial vertical coordinate values will be corrected first due to their low accuracy. That is, for each sample reference object, the initial horizontal coordinate value of the initial pixel coordinate is substituted into the straight-line equation of the first fitted straight line where the sample reference object is located to obtain the intermediate second coordinate value. The intermediate second coordinate value is the corrected vertical coordinate value corresponding to the initial vertical coordinate value of the initial pixel coordinate of the sample reference object.
For each sample reference object, the initial first coordinate value and the intermediate second coordinate value constitute the intermediate pixel coordinate. For example, the initial pixel coordinates of the twenty sample reference objects are (x1, y1)˜(y20, y20), and after correction as described above, the intermediate pixel coordinates (x1, y1′)˜(x20, y20′) of the twenty sample reference objects are obtained.
Straight-line fitting can be performed on the sample reference objects on each straight line in the second direction respectively based on the intermediate pixel coordinates of the sample reference objects to obtain the multiple second fitted straight lines as follows. Straight-line fitting is performed on the sample reference objects on each straight line in the second direction respectively based on the initial first coordinate values and the intermediate second coordinate values in the intermediate pixel coordinates of the sample reference objects to obtain the multiple second fitted straight lines.
Line segments between sample reference objects on one second fitted straight line do not intersect line segments between sample reference objects on another second fitted straight line. Line segments between sample reference objects on each second fitted straight line intersect line segments between sample reference objects on multiple first fitted straight lines.
After straight-line fitting is performed on the sample reference objects on each straight line in the second direction respectively based on the intermediate pixel coordinates of the sample reference objects, line segments between sample reference objects on one second fitted straight line do not intersect line segments between sample reference objects on another second fitted straight line. For example, if first straight-line fitting is performed on the sample reference objects to obtain multiple first fitted straight lines corresponding to the straight lines L in
For example, second straight-line fitting can be performed to obtain the multiple second fitted straight lines as follows. The sample reference objects in the sample reference object array can be divided into multiple groups of sample reference objects. Each group of sample reference objects belong to a same straight line in the world coordinate system. In one case, when first straight-line fitting is performed, the sample reference objects are grouped according to which straight line L they belong to in the world coordinate system. Then, when second straight-line fitting is performed, the sample reference objects are grouped according to which straight line H they belong to in the world coordinate system. In another case, when first straight-line fitting is performed, the sample reference objects will be grouped according to which straight line H they belong to in the world coordinate system. Then, when second straight-line fitting is performed, the sample reference objects will be grouped according to which straight line L they belong to in the world coordinate system.
For example, to obtain multiple first fitted straight lines, the sample reference objects are grouped according to which straight line L they belong to in the world coordinate system. Then, to obtain multiple second fitted straight lines, the sample reference objects are grouped according to which straight line H they belong to in the world coordinate system. That is, sample reference objects belonging to a same straight line H in the world coordinate system are divided into one group. For example, for the sample image shown in
For each group of sample reference objects, second straight-line fitting can be performed with the intermediate pixel coordinates of the group of sample reference objects by adopting the least square method. A second straight-line equation corresponding to a second fitted straight line can be obtained according to the following equations (4) -(6):
where (xi, yi′) indicates an intermediate pixel coordinate of the i-th sample reference object in the group of sample reference objects, the intermediate pixel coordinate includes an initial horizontal coordinate value and an intermediate vertical coordinate value, n indicates the number of sample reference objects in the group,
For each group of sample references objects, after the intermediate pixel coordinates corresponding to the group of sample reference objects are substituted into equations (4) -(6), the unknown parameters b2 and b3 in the second straight-line equation corresponding to the group of sample reference objects can be obtained. In this way, the second straight-line equation, y=b3x-b2, of the second fitted straight line corresponding to the group of sample reference objects can be obtained.
At S403, the corrected pixel coordinates are obtained based on the multiple first fitted straight lines and the multiple second fitted straight lines.
Pixel coordinates corresponding to intersections of the multiple first fitted straight lines and the multiple second fitted straight lines can be assigned as the corrected pixel coordinates.
The corrected pixel coordinates of the sample reference objects in the sample reference object array can be obtained by calculating coordinate values of the intersections of the first fitted straight lines and the second fitted straight lines with corresponding first straight-line equations and second straight-line equations. For example, the intermediate pixel coordinates of the twenty sample reference objects are (x1, y1′)˜(x20, y20′), and after correction as described above, the corrected pixel coordinates (xi″, yi″)˜(x20, y20′) of the twenty sample reference objects are obtained.
According to S401-S403, the initial pixel coordinates of the sample reference objects can be corrected to obtain the corrected pixel coordinates as follows. The initial pixel coordinates of the sample reference objects can be corrected based on different straight lines to which the sample reference objects belong. For example, straight lines in two different directions are selected for correcting the initial pixel coordinates of the sample reference objects to obtain the corrected pixel coordinates with high accuracy. During the correction of the initial pixel coordinates of the sample reference objects, for each initial pixel coordinate, one coordinate value in the initial pixel coordinate is corrected, and then the other coordinate value in the initial pixel coordinate is corrected. In this way, a corrected coordinate with high accuracy can be obtained.
As for S104, after the corrected pixel coordinates of the sample reference objects in the image coordinate system are obtained, the homography matrix of the image acquisition device can be determined based on the world coordinates of the sample reference objects in the world coordinate system and the corrected pixel coordinates of the sample reference objects in the image coordinate system. For example, a pixel coordinate matrix of the corrected pixel coordinates of the sample reference objects in the image coordinate system can be constructed. A world coordinate matrix of the world coordinates of the sample reference objects in the world coordinate system can be constructed. Then, the pixel coordinate matrix and world coordinate matrix known as well as the homography matrix of the image acquisition device unknown are substituted into a conversion equation for the image acquisition device between pixel coordinates and world coordinates to determine the homography matrix of the image acquisition device.
For example, the world coordinates of the sample reference objects in the sample reference object array in the world coordinate system are denoted as (X1, Y1)˜(Xn, Yn), the world coordinate matrix is denoted as A, the pixel coordinate matrix is denoted as C, and the homography matrix is denoted as B, which are expressed as follows:
Then, the world coordinate matrix A, the pixel coordinate matrix C, and the homography matrix B are substituted into the conversion equation for the image acquisition device between pixel coordinates and world coordinates. The conversion equation is expressed as the following equation (7):
A=B×C (7).
The conversion equation is solved to obtain the homography matrix of the image acquisition device as follows: B=(AAT))*(CAT)−1.
In implementations of the present disclosure, the initial pixel coordinates of the sample reference objects in the image coordinate system can be corrected to obtain the corrected pixel coordinates with high accuracy of the sample reference objects in the image coordinate system. Then, according to the world coordinates of the sample reference objects in the sample reference object array in the world coordinate system and the corrected pixel coordinates of the sample reference objects in the image coordinate system, an accurate homography matrix of the image acquisition device can be obtained. In this way, the accuracy of calibration for the image acquisition device can be improved.
Further, as shown in
At S501, multiple test images are acquired with the image acquisition device after the homography matrix of the image acquisition device is determined.
This image acquisition device is the same type as the image acquisition device mentioned above, and shooting angles for the multiple test reference object arrays are the same as that for the sample reference object array mentioned above.
The process of setting test reference objects is similar to sample reference objects, which will not be repeated here. Multiple different test reference object arrays can be set, so that the image acquisition device can shoot for each test reference object array to obtain multiple test images.
At S502, for each test image, test pixel coordinates of test reference objects in the test image in the image coordinate system are determined.
A method for determining the test pixel coordinates of the test reference objects in the test image in the image coordinate system here is the same as the method for determining the corrected pixel coordinates of the sample reference objects in the sample image in the image coordinate system described above, which will not be repeated.
At S503, test world coordinates of the test reference objects in the world coordinate system are determined based on the test pixel coordinates and the homography matrix.
For each test image, after the test pixel coordinates of the test reference objects are obtained, horizontal coordinate values and vertical coordinate values in the test pixel coordinates constitute a test pixel coordinate matrix. The test pixel coordinate matrix and the homography matrix are input to the conversion equation for the image acquisition device between pixel coordinates and world coordinates to obtain the test world coordinates of the test reference objects in the test image in the world coordinate system.
At S504, an accuracy of the homography matrix is determined based on real-world coordinates and the test world coordinates of the test reference objects in the multiple test images.
For each test image, the real-world coordinates of the test reference objects are compared with the test world coordinates respectively to determine whether the test world coordinates of the test reference objects in the test image are accurate. A ratio of the number of test reference objects with accurate test world coordinates to the total number of test reference objects is assigned as the accuracy of the homography matrix.
In implementations of the present disclosure, the accuracy of the homography matrix is checked to determine whether the accuracy of the obtained homography matrix meets a set condition. When the accuracy of the homography matrix does not meet the set condition, the homography matrix can be corrected in time. For example, the calibration process for the image acquisition device will be executed again, that is, operations of S101-S104 will be executed again to obtain a homography matrix with higher accuracy, thereby ensuring that the ranging with the image acquisition device is accurate.
Further, after the homography matrix of the image acquisition device is determined, a location of a target object can be determined based on the homography matrix as follows.
At S601, a target image of a target object is acquired with an image acquisition device.
At S602, a pixel coordinate of the target object in an image coordinate system is determined based on the target image.
At S603, a world coordinate of the target object in a world coordinate system is determined based on the pixel coordinate and a homography matrix of the image acquisition device.
At S604, a distance between the target object and the preset location is determined based on the world coordinate of the target object in the world coordinate system and a coordinate of a preset location in the world coordinate system.
Taking a vehicle as an example, the preset location can be at the projection of the center of a front axle of the vehicle on the ground, or at the projection of the center of the body of the vehicle on the ground. When the preset location is set as the origin, a world coordinate of the origin in the world coordinate system is known. The preset location can be used as a vehicle ranging point for ranging between the target object and the vehicle.
After the homography matrix of the image acquisition device is obtained, operations of S601-S604 are executed for ranging with the homography matrix. The target object in the target image has an area. After the target image is obtained, a ranging point of the target object can be determined according to the target image. Then, a distance between the target object and the vehicle can be determined based on world coordinates of the ranging point and the preset location in the world coordinate system.
In some examples, after the target image of the target object is obtained, a marking box for the target object can be obtained based on image recognition technologies. During the calibration for the image acquisition device, to determine the homography matrix of the image acquisition device, the location where a selected cone reference object is tangent to the ground is determined as the location of the reference object. Similarly, the ranging point of the target object should be selected as a point on a tangent line between the marking box and the ground in the target image. For example, a center point of the tangent line between the marking box and the ground can be selected as the ranging point. Then, a pixel coordinate of the ranging point can be used as the pixel coordinate of the target object in the image coordinate system.
After the pixel coordinate of the target object in the image coordinate system is obtained, the pixel coordinate of the target object in the image coordinate system and the homography matrix are input into the conversion equation for the image acquisition device between pixel coordinates and world coordinates to obtain the world coordinate of the target object in the world coordinate system. According to the world coordinate of the target object in the world coordinate system and the world coordinate of the preset location, a Euclidean distance between the target object and the preset location can be calculated to determine the distance between the target object and the vehicle.
In implementations of the present disclosure, after the homography matrix with high accuracy is obtained, the homography matrix can be used to accurately determine the world coordinate of the target object in the world coordinate system, and then the distance to the target object.
In conclusion, according to the calibration method provided, after the sample image of the sample reference objects is acquired with the image acquisition device, the initial pixel coordinates of the sample reference objects in the image coordinate system are determined. Straight-line fitting is performed on the sample reference object array in the sample image based on the initial pixel coordinates of the sample reference objects in the image coordinate system. The initial pixel coordinates are corrected based on the fitted straight lines to obtain the corrected pixel coordinates of the sample reference objects in the image coordinate system.
The sample reference objects can be placed in advance. For example, the sample reference objects can be arranged in a sample reference object array, so that sample reference objects in a same row or column are on a same straight line in the world coordinate system. The initial pixel coordinates of the sample reference objects in the image coordinate system can be corrected by performing straight-line fitting with the initial pixel coordinates of the sample reference objects in the image coordinate system to obtain the corrected pixel coordinates with high accuracy of the sample reference objects in the image coordinate system. Then, according to the world coordinates of the sample reference objects in the sample reference object array in the world coordinate system and the corrected pixel coordinates of the sample reference objects in the image coordinate system, an accurate homography matrix of the image acquisition device can be obtained. In this way, the accuracy of calibration for the image acquisition device can be improved.
It can be understood by those of ordinary skill in the art that, in the method in the implementations, the sequence of each step does not refer to a strict execution sequence and is not intended to limit the implementation process and a specific execution sequence of each step should be determined by functions and probable internal logic thereof.
Based on the same concept, a calibration apparatus corresponding to the calibration method is provided according to implementations of the present disclosure. The principle of the calibration apparatus to solve the problem is similar to that of the calibration method. Therefore, implementations of the calibration apparatus can refer to implementations of the calibration method, which will not be repeated here.
In an implementation, the coordinate correction module 703 is configured to: perform straight-line fitting on the sample reference objects on each straight line in a first direction respectively based on the determined initial pixel coordinates of the sample reference objects in the image coordinate system to obtain a multiple first fitted straight lines, correct the initial pixel coordinates of the sample reference objects in the image coordinate system based on the multiple first fitted straight lines to obtain intermediate pixel coordinates, and perform straight-line fitting on the sample reference objects on each straight line in a second direction respectively based on the intermediate pixel coordinates of the sample reference objects to obtain multiple second fitted straight lines, where each straight line in the first direction intersects each straight line in the second direction, and obtain the corrected pixel coordinates based on the multiple first fitted straight lines and the multiple second fitted straight lines.
In an implementation, each initial pixel coordinate includes an initial first coordinate value and an initial second coordinate value, and a first coordinate axis corresponding to the initial first coordinate value is perpendicular to a second coordinate axis corresponding to the initial second coordinate value. The coordinate correction module 703 configured to correct the initial pixel coordinates of the sample reference objects in the image coordinate system based on the multiple first fitted straight lines to obtain the intermediate pixel coordinates is configured to: for each sample reference object, substitute the initial first coordinate value of the initial pixel coordinate into a straight-line equation of the first fitted straight line where the sample reference object is located to obtain an intermediate second coordinate value, where the intermediate pixel coordinate of the sample reference object includes the initial first coordinate value and the intermediate second coordinate value of the sample reference object. The coordinate correction module 703 configured to perform straight-line fitting on the sample reference objects on each straight line in the second direction respectively based on the intermediate pixel coordinates of the sample reference objects to obtain the multiple second fitted straight lines is configured to: perform straight-line fitting on the sample reference objects on each straight line in the second direction respectively based on the initial first coordinate values and the intermediate second coordinate values in the intermediate pixel coordinates of the sample reference objects to obtain the multiple second fitted straight lines.
In an implementation, the coordinate correction module 703 configured to obtain the corrected pixel coordinates based on the multiple first fitted straight lines and the multiple second fitted straight lines is configured to: assign pixel coordinates corresponding to intersections of the multiple first fitted straight lines and the multiple second fitted straight lines as the corrected pixel coordinates.
In an implementation, the first coordinate axis is a horizontal axis of the image coordinate system and the second coordinate axis is a vertical axis of the image coordinate system, or the first coordinate axis is the vertical axis of the image coordinate system and the second coordinate axis is the horizontal axis of the image coordinate system.
In an implementation, the second determination module 704 is further configured to: acquire multiple test images with the image acquisition device after determining the homography matrix of the image acquisition device, determine, for each test image, test pixel coordinates of test reference objects in the test image in the image coordinate system, determine test world coordinates of the test reference objects in the world coordinate system based on the test pixel coordinates and the homography matrix, and determine an accuracy of the homography matrix based on real-world coordinates and the test world coordinates of the test reference objects in the multiple test images.
As shown in
In an implementation, the second determination module 803 is further configured to: determine a distance between the target object and the preset location based on the world coordinate of the target object in the world coordinate system and a coordinate of a preset location in the world coordinate system after determining the world coordinate of the target object in the world coordinate system.
An electronic device 900 corresponding to the calibration method as shown in
An electronic device 1000 corresponding to the location determination method as shown in
A computer-readable storage medium is provided according to implementations of the present disclosure. The computer-readable storage medium is configured to store computer programs. When executed by a processor, the computer programs are configured to perform operations of the calibration method the location determination method according to implementations of the present disclosure.
A computer program product is provided for the calibration method or the location determination method according to implementations the present disclosure. The computer program product includes a computer-readable storage medium configured to store program codes. Instructions in the program codes can be used to perform operations of the calibration method or the location determination method, specifically referring to the method implementations, which will not be repeated here.
It can be clearly understood by those of ordinary skill in the art that, for convenience and brevity, specific operation processes of systems and apparatuses above may refer to the corresponding processes in method implementations, which will not be repeated here. In some implementations of the present disclosure, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in another manner. Apparatus implementations described above are only schematic. For example, division of units is only logic function division, and other division manners may be adopted during practical implementation. For another example, multiple units or components may be combined or integrated into another system, or some features may be emitted or not executed. In addition, coupling or direct coupling or communication connection between displayed or discussed components may be indirect coupling or communication connection through some communication interfaces between apparatuses or units, and may be electrical, mechanical, or other forms.
Units described as separate parts may or may not be physically separated, and parts displayed as units may or may not be physical units, and namely may be in the same place, or may be distributed to multiple network units. Part or all of the units may be selected to achieve the purpose of the solutions of the implementations according to a practical requirement.
In addition, functional units in implementations of the present disclosure may be integrated into a processing unit, each unit may also be physically independent, or two or more units may be integrated into one unit.
When realized in form of software functional units and sold or used as independent products, the functions may be stored in a non-volatile computer-readable storage medium executable by the processor. Based on such an understanding, the technical solutions of the present disclosure substantially or parts making contributions to the related art or parts of the technical solutions may be embodied in form of software product, and the computer software product is stored in a storage medium, including multiple instructions configured to enable a computer device, which may be a personal computer, a server, a network device, or the like, to execute all or part of the operations of the methods in implementation of the present disclosure. The storage medium includes various media capable of storing program codes such as a U disk, a mobile hard disk, a read-only memory (ROM), a random-access memory (RAM), a magnetic disk, or an optical disk.
Finally, it should be noted that the above-mentioned implementations are only examples of the present disclosure, which are used to illustrate the technical solutions of the present disclosure but not to limit, and the scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing implementations, it should be understood by those of ordinary skill in the art that, without departing from the technical scope of the present disclosure, any person of ordinary skill in the art can modify or change the technical solutions described in the foregoing implementations, or replace some of the technical features with equivalent technical features. However, modifications, changes, or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the implementations of the present disclosure, and should be covered within the scope of the present disclosure. Therefore, the scope of the present disclosure shall be subject to the scope of the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202010175090.3 | Mar 2020 | CN | national |
This application is a continuation under 35 U.S.C. § 120 of International Application No. PCT/CN2020/142509, filed on Dec. 31, 2020, which claims priority under 35 U.S.C. § 119(a) and/or PCT Article 8 to Chinese Patent Application No. 202010175090.3, filed on Mar. 13, 2020, the disclosures of which are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2020/142509 | Dec 2020 | US |
| Child | 17502387 | US |
