1. Field of the Invention
The invention relates to a technology that derives data relating to a camera.
2. Description of the Background Art
A phenomenon called a distortion in which an image of a subject is distorted in a captured image obtained by a camera, occurs. Especially, in the captured image obtained by a camera using a fish lens, a relatively large distortion occurs.
A characteristic of such distortion in the captured image obtained by a camera (hereinafter referred to as “distortion characteristic”) can be derived by, for example, Zhang's method. The Zhang's method is disclosed in “A flexible new technique for camera calibration” in IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11): 1330-1334, 2000. In the Zhang's method, a camera captures an image of many marks arrayed in a grid and then a distortion characteristic is derived based on positions of images of the many marks in the captured image.
A distortion characteristic of the camera can be expressed by a relation between an angle of incidence of light from the subject and an image height (a distance from a center of a picture image) of the subject. Such a distortion characteristic of the camera is defined in advance from design. However, an actual distortion characteristic of the camera varies depending on the camera due to, for example, a manufacturing error, and is different from a distortion characteristic in the design.
Recently, a technology that generates a combined image (overhead image) showing surroundings of the vehicle viewed from a virtual point of view by combining plural captured images obtained by plural cameras installed on/in a vehicle, has become popular. In a case of generating such a combined image, a distortion in an image of a subject in the captured image is corrected taking the distortion characteristic into consideration. However, an actual distortion characteristic of a camera is different from a distortion characteristic in design. Therefore, in a case of simply using the data showing the distortion characteristic in design, a distortion of a subject image is not appropriately corrected. As a result, the generated combined image has a problem, for example, where images of a same subject are misaligned at a seam between the captured images.
It is required to derive the actual distortion characteristic of each of the plural cameras by using a method such as the aforementioned Zhang method, to solve such a problem. However, since there are many restrictions on equipment placed in a work area of a vehicle factory, it is difficult to actually place many marks in a grid in such a work area. Therefore, it was required to find a technology that derives an actual distortion characteristic of a camera by using more simple equipment.
According to one aspect of the invention, a data deriving apparatus derives data relating to a camera. The apparatus includes a controller configured to: identify positions of two mark images in a captured image of two marks obtained by the camera; derive an error rate of an actual image height of a subject image in the captured image to a design image height based on a first distortion characteristic defined in advance from design of the camera, based on the positions of the two mark images; and derive an actual second distortion characteristic of the camera, based on the error rate and the first distortion characteristic.
Since the actual second distortion characteristic of the camera is derived based on the positions of the two mark images in the captured image, the actual second distortion characteristic of the camera can be derived with relatively simple equipment.
According to another aspect of the invention, the controller derives the error rate based on a result of a comparison of an actual distance between the two marks with a derived distance between the two marks derived based on the positions of the two mark images in the captured image.
The actual second distortion characteristic of the camera can be derived by a relatively simple algorithm.
Moreover, according to another aspect of the invention, the controller obtains an installation parameter relating to installation of the camera, based on the second distortion characteristic and the captured image of the two marks obtained by the camera.
The installation parameter can be obtained with precision by using the actual second distortion characteristic of the camera.
Therefore, the objective of the invention is to derive an actual distortion characteristic of a camera with relatively simple equipment.
These and other objects, features, aspects and advantages of the invention will become more apparent from the following detailed description of the invention when taken in conjunction with the accompanying drawings.
An embodiment of the invention is hereinafter explained with reference to the drawings.
<1. System Outline>
The actual distortion characteristic of each of the cameras 5 is different from a distortion characteristic defined in advance from design (hereinafter referred to as “design characteristic”). Therefore, the calibration system 10 obtains the actual data showing the actual distortion characteristics of the cameras 5. Moreover, an optical axis of each of the cameras 5 is slightly different from a direction defined from design. Therefore, the calibration system 10 obtains the installation parameter relating to the installation of each of the cameras 5, such as a horizontal angle, a vertically-tilted angle, and a rotated angle.
The calibration system 10 includes the plurality of cameras 5 to be installed on the vehicle 9, a vehicle-mounted apparatus 2, and four position markers 7 located separately from the vehicle 9. The four position markers 7 are located in predetermined positions in a work area of a vehicle factory, a vehicle maintenance shop or another place where the calibration process is performed.
As shown in
In a case of performing the calibration process, as shown in
Based on the captured images obtained in a manner mentioned above, the vehicle-mounted apparatus 2 functions as a data deriving apparatus that derives data of the cameras 5. Based on the captured images, the vehicle-mounted apparatus 2 derives the actual data showing actual distortion characteristics of the cameras 5 and the installation parameters of the cameras 5. The vehicle-mounted apparatus 2 identifies positions of images of the marks 70 included in the captured images and then derives the actual data and the installation parameters based on the positions of the images of the marks 70.
Such a calibration process is performed when the plurality of cameras 5 get installed on the vehicle 9. The actual data and the installation parameters obtained in the calibration process are stored in the vehicle-mounted apparatus 2 and are used in an image processing performed by the vehicle-mounted apparatus 2 after the storage.
<2. Vehicle-Mounted Camera>
The plurality of cameras 5 includes a front camera 5F, a back camera SB, a left side camera 5L, and a right side camera SR. Each of these four cameras 5F, 5B, 5L, and 5R is provided on a position different from one another and captures the image of the surroundings of the vehicle 9 in a direction different from one another.
The front camera 5F is provided in a vicinity of a horizontal center of a front end of the vehicle 9, having an optical axis 5Fa in a front direction (a straight ahead direction) of the vehicle 9. The back camera 5B is provided in a vicinity of a horizontal center of a rear end of the vehicle 9, having an optical axis 5Ba in a back direction (a direction opposite to the straight ahead direction) of the vehicle 9. The left side camera 5L is provided on a left side mirror 93L of the vehicle 9, having an optical axis 5La in a left direction (a direction orthogonal to the straight ahead direction) of the vehicle 9. Moreover, the right side camera 5R is provided on a right side mirror 93R of the vehicle 9, having an optical axis 5Ra in a right direction (a direction orthogonal to the straight ahead direction) of the vehicle 9.
A fish-eye lens is used for a lens of each of the plurality of cameras 5, and each of the cameras 5 has an angle of view a of 180 degrees or more. Therefore, it is possible to capture images covering 360-degree surroundings of the vehicle 9 by using the four cameras 5F, 5B, 5L, and 5R. Each of the regions A1, A2, A3, and A4 of the left region in front of the vehicle 9, the right region in front of the vehicle 9, the left region behind the vehicle 9, and the right region behind the vehicle 9 is a photographic subject redundantly of two of the four cameras 5. Each of the four position markers 7 is placed in each of the four regions A1, A2, A3, and A4 that is the photographic subject of two of the cameras 5 as mentioned above. (Refer to
<3. Vehicle-Mounted Apparatus>
The vehicle-mounted apparatus 2 includes a display 26, an operation part 25, an image obtaining part 22, an image combiner 23, a storage 24, and a controller 21.
The display 26 is a thin display apparatus including, for example, a liquid crystal panel, and shows a variety of information and images. The operation part 25 is, for example, an operation button that receives a user operation. When a user operates the operation part 25, a signal representing a content of the operation is input to the controller 21.
The image obtaining part 22 obtains the captured images obtained by the four cameras 5, from the four cameras 5. The image obtaining part 22 includes a basic image processing function such as A/D conversion function that converts an analog captured image into a digital captured image. The image obtaining part 22 performs a predetermined image processing on the obtained captured images and inputs the processed captured images to the image combiner 23 and the controller 21.
The image combiner 23 is, for example, a hardware circuit, and performs a predetermined image processing. The image combiner 23 generates the combined image (overhead view) showing the surroundings of the vehicle 9 viewed from an arbitrary virtual point of view by using the four captured images respectively captured by the four cameras 5. A method that the image combiner 23 uses to generate the combined image viewed from the virtual point of view will be described later.
The storage 24 is, for example, a nonvolatile memory such as a flash memory, and stores various types of information. The storage 24 stores design data 24a, actual data 24b, and a installation parameter 24c. Moreover, the storage 24 stores a program serving as firmware of the vehicle-mounted apparatus 2.
The design data 24a is distortion data showing the design characteristic of the cameras 5. On the other hand, the actual data 24b is distortion data showing the actual distortion characteristic of each of the cameras 5. The design data 24a is common data for all the four cameras 5F, 5B, 5L, and 5R. However, the actual data 24b varies depending on each of the cameras 5. The actual data 24b is obtained in the calibration process based on the design data 24a. Therefore, before the calibration process, the design data 24a is stored in the storage 24, but the actual data 24b is not stored.
Moreover, the installation parameter 24c is a parameter relating to the installation of each of the cameras 5. The installation parameter 24c includes a parameter that indicates a direction of the optical axis of each of the cameras 5 such as the horizontal angle, the vertically-tilted angle, and the rotated angle. The installation parameter 24c varies depending on each of the cameras 5. The installation parameter 24c is also obtained in the calibration process. Therefore, before the calibration process, the installation parameter 24c is not stored in the storage 24.
The storage 24 stores the actual data 24b and the installation parameter 24c corresponding to each of the four cameras 5F, 5B, 5L, and 5R. The actual data 24b and the installation parameter 24c are used when the image combiner 23 generates the combined image.
The controller 21 is a microcomputer that comprehensively controls the entire vehicle-mounted apparatus 2. The controller 21 includes a CPU, a RAM, a ROM, etc. Various functions of the controller 21 are implemented by arithmetic processing performed by the CPU in accordance with a program stored in the storage 24. A position identifying part 21a, an error rate deriving part 21b, a distortion deriving part 21c, and a parameter obtaining part 21d, shown in
<4. Generation of the Combined Image>
Next explained is a method used when the image combiner 23 generates the combined image.
When capturing images, the front camera 5F, the back camera 513, the left side camera 5L, and the right side camera 5R of the vehicle-mounted apparatus 2 obtain four captured images GF, GB, GL, and GR respectively that show regions in front of, behind, on left of and on right of the vehicle 9 respectively. Images of subjects existing in the 360-degree surroundings of the vehicle 9 are included in the four captured images GF, GB, GL, and GR.
The image combiner 23 projects data (values of individual pixels) included in the four captured images GF, GB, GL, and GR onto a projection plane TS that is a 3D curved plane in a virtual 3D space. The projection plane TS is substantially shaped like a hemisphere (a bowl). A center area of the projection plane TS (a bottom of the bowl) is defined as an area for the vehicle 9. On the other hand, each of areas other than the center area of the projection plane TS corresponds to one of the captured images GF, GB, GL, and GR. The image combiner 23 projects the data included in the captured images GF, GB, GL, and GR onto the areas other than the center area.
As shown in
In reference back to
Next, the image combiner 23 sets a virtual point of view VP to the 3D space. The image combiner 23 is capable of setting the virtual point of view VP in an arbitrary direction of view at an arbitrary position in the 3D space. Then, the image combiner 23 clips out, as an image, from the projection plane TS, an area within a range of a predetermined angle of view from the set virtual point of view VP. Moreover, the image combiner 23 renders the polygonal model in accordance with the set virtual point of view VP and then superimposes a two-dimensional vehicle image 90, which is a result of the rendering, on the clipped-out image. Thus, the image combiner 23 generates a combined image CP showing the vehicle 9 and a vicinity region of the vehicle 9 viewed from the virtual point of view VP.
For example, as shown in
As mentioned above, the actual distortion characteristic of each of the cameras 5 is different from the design characteristic. Therefore, if the design data 24a showing the design characteristic is simply used to correct the distortions of the subject images, the distortions of the subject images are not corrected appropriately. As a result, a problem where images of a same subject are misaligned at a seam B (refer to
Moreover, a target region in the captured image G including data to be projected on the projection plane TS varies in accordance with an error caused during installation of the camera 5 that has obtained the captured image G. Therefore, the image combiner 23 corrects the target region that is projected on the projection plane TS, using the installation parameters 24c (the horizontal angle, the vertically-tilted angle, the rotated angle, etc.) of the camera 5.
If there is no error caused during the installation of the camera 5 that has obtained the captured image G shown in
In the manner mentioned above, the image combiner 23 can generate an appropriate combined image CP by using the actual data 24b and the installation parameters 24c of the camera 5 when generating the combined image CP.
<5. Calibration Process>
Next explained is the calibration process for obtaining the actual data 24b and the installation parameters 24c of the cameras 5. The calibration process is performed by an operator who implements a predetermined operation with the operation part 25 in a state where the vehicle 9 is located in the predetermined position in the work area having the four position markers 7 located beforehand, as shown in
First, the controller 21 selects one from amongst the four cameras 5 as “a target camera 5” for the process (a step S11). The actual data 24b and the installation parameters 24c of the target camera 5 are obtained in steps from a step S12 to a step S14 in the process.
In the step S12, the target camera 5 obtains the captured image by capturing an image of the vicinity region of the vehicle 9 including the marks 70 on the two position markers 7.
Next, in a step S13, a distortion derivation process is performed to derive the actual data 24b showing the actual distortion characteristic of the target camera 5. The actual data 24b is derived in the distortion derivation process, based on positions of the two mark images 71a and 72a included in the captured image GC. The distortion derivation process is performed by the position identifying part 21a, the error rate deriving part 21b, and the distortion deriving part 21c. Details performed by the parts will be explained later. The actual data 24b obtained in the distortion derivation process is associated with the target camera 5 and is stored in the storage 24.
Next, in the step S14, the parameter obtaining part 21d performs a parameter obtaining process to obtain the installation parameters 24c relating to installation of the target camera 5. The parameter obtaining part 21d first corrects distortions of subject images in the captured image GC, using the actual data 24b obtained in the step S13. Then the parameter obtaining part 21d derives the installation parameters 24c of the target camera 5 based on the positions of the two mark images 71a and 72a included in the captured image GC, by using a common method such as a method disclosed in Japanese Patent Application Laid-open Publication No. 4803449. For example, the parameter obtaining part 21d can derive the horizontal angle based on horizontal positions of the two mark images 71a and 72a in the captured image GC, the vertically-tilted angle based on vertical positions of the two mark images 71a and 72a in the captured image GC, and the rotated angle based on a difference in height between the two mark images 71a and 72a in the captured image GC.
As mentioned above, after correcting the distortions of the subject images in the captured image GC by using the actual data 24b showing the actual distortion characteristic of the target camera 5, the parameter obtaining part 21d derived the installation parameters 24c based on the captured image GC. Therefore, the installation parameters 24c can be obtained with precision. The installation parameters 24c obtained in the parameter obtaining process are associated with the target camera 5 and are stored in the storage 24.
When the parameter obtaining process is completed, the controller 21 determines whether or not the actual data 24b and the installation parameters 24c of all the four cameras 5 have been derived. If there is any camera 5 of which the actual data 24b and the installation parameters 24c have not been derived (No in a step S15), the camera 5 that has not been set as the target camera 5 is set as a new target camera 5, and the process mentioned above is repeated. By repeating the process mentioned above, the actual data 24b and the installation parameters 24c of all the four cameras 5 are derived. When the actual data 24b and the installation parameters 24c of all the four cameras 5 are derived (Yes in the step S15), the calibration process ends.
<6. Distortion Derivation Process>
Next, the distortion derivation process in the step S13 is explained in detail. First explained is a relation between the design data 24a showing the design characterisitic and the actual data 24b showing the actual distortion of the camera 5.
As shown in
An error rate err (%) of the actual image height r′ to the design image height r is herein defined by a following mathematical formula 1.
Moreover, when f represents a focal length of the camera 5, the design image height r is defined by, for example, a following mathematical formula 2.
r=2f·tan(θ/2) [Mathematical formula 2]
On the other hand, the actual distortion characteristic is different from the design characteristic due to a manufacturing error relating to a distance (equal to the focal length f) between the lens of the camera 5 and the image sensor. Therefore, when k represents an error of the focal length f, the actual image height r′ can be represented by a following mathematical formula 3.
r′=2(f+k)·tan(θ/2) [Mathematical formula 3]
Thus, the error rate err of the image height represented by the mathematical formula 1 does not depend on the angle of incidence θ and does not vary as shown in
Moreover, in the distortion derivation process, a camera coordinate system set based on the target camera 5 is used.
An origin Ow of the world coordinate system is different from an origin Oc of the camera coordinate system. Furthermore, generally, directions of the individual axes of the world coordinate system are different from directions of the individual axes of the camera coordinate system. The origin Oc of the camera coordinate system in the world coordinate system (i.e. a position of the target camera 5 relative to the vehicle 9) is known. In addition, the Zc axis of the camera coordinate system is directed along the optical axis of the target camera 5.
As mentioned above, the position markers 7 are located at the predetermined positions and the positions of the position markers 7 relative to the vehicle 9 are unvaried. Therefore, coordinate positions of the marks 71 and 72 in the world coordinate system are known. Therefore, a distance d1 from the target camera 5 to the mark 71 on the left side, a distance d2 from the target camera 5 to the mark 72 on the right side, an actual distance m between the mark 71 and the mark 72 can be derived from three coordinate positions of the target camera 5, the mark 71 and the mark 72 in the world coordinate system. The distances d1, d2, and m are stored in the storage 24 beforehand and are used in the distortion derivation process. However, the distances d1, d2 and m may be derived in a step in the distortion derivation process based on the three coordinate positions.
First, the position identifying part 21a identifies positions of the two mark images 71a and 72a in the captured image GC (a step S 21). The position identifying part 21a detects the positions of the two mark images 71a and 72a in the captured image GC like an upper drawing in
Next, the error rate deriving part 21b derives the error rate err based on the positions P1 and P2 of the two mark images 71a and 72a in the captured image GC. The error rate deriving part 21b sets a temporary value to the error rate err to be derived in this process. Then the error rate deriving part 21b derives an evaluation value E by performing a series of steps from a step S23 to a S27, using the temporary value set to the error rate err.
The error rate deriving part 21b changes the temporary value set to the error rate err (%) by 0.1 in a range from −5.0 to +5.0 (steps S22, S28, and S29). Every time when changing the temporary value for the error rate err, the error rate deriving part 21b repeats the series of the steps from the step S23 to S27. Thus, the error rate deriving part 21b derives the evaluation value E per the temporary value set to the error rate err.
In the step S23, the design image height r for each of the two mark images 71a and 72a is derived by using the error rate en for which temporary value has been set.
First, by deriving distances from an image center (a position of the optical axis) of the captured image GC to the respective positions P1 and P2 identified in the step S21, as shown in the mid drawing in
Therefore, as shown in the lower drawing in
Next, in a step S24 (refer to
First, in reference to the design data 24a (the solid-line curve in
In the captured image GC, herein, Φ1 is an angle between a horizontal line and a straight line from the image center to the position P1, and Φ2 is an angle between the horizontal line and a straight line from the image center to the position P2 (refer to
As mentioned above, the optical axis 5a of the target camera 5 is directed along the Zc axis in the camera coordinate system, and the position of the target camera 5 is the origin of the camera coordinate system. Therefore, as shown in a following mathematical formula 5, the eye vector v1 (refer to
{right arrow over (v)}
1=(sin θ1·sin Φ1, sin θ1·cos Φ1, cos θ1)
{right arrow over (v)}
2=(sin θ2·sin Φ2, sin θ2·cos Φ2, cos θ2) [Mathematical formula 5]
Next, in a step S25 (refer to
First, as shown in
{right arrow over (T)}
1
=d
1
·{right arrow over (v)}
1
{right arrow over (T)}
2
=d
2
√{right arrow over (v)}
2 [Mathematical formula 6]
Then, a vector T3, shown in
{right arrow over (T)}
3
={right arrow over (T)}
1
−{right arrow over (T)}
2 [Mathematical formula 7]
Moreover, a length of the vector T3 is derived as a distance M between the two marks 71 and 72 (hereinafter referred to as “derived distance”) by a mathematical formula 8.
M=∥{right arrow over (T)}
3∥ [Mathematical formula 8]
Next, in a step S26 (refer to
E=|M−m|
The evaluation value E represents an accuracy level of the temporary value set to the error rate err. The smaller the evaluation value E is (i.e. the smaller the difference between the derived distance M and the actual distance m is), the closer to an actual error rate err the temporary value is.
Next, in a step S27, the evaluation value E derived for a current temporary value for the error rate err is compared to a minimum evaluation value among the evaluation values E derived before. In a case where the current evaluation value E is smaller than the minimum evaluation value among the evaluation values E derived before, the minimum evaluation value is updated to set the current evaluation value E as a new minimum evaluation value.
The series of the steps from the S23 to the S27 are repeated. In a case where the process relating to all the temporary values to be set to the error rate err is completed (Yes in the step S28), the temporary value corresponding to the minimum evaluation value E at that time is a closest to the actual error rate err. Therefore, the error rate deriving part 21b derives the temporary value as the actual error rate err (a step S30).
When the error rate deriving part 21b derives the error rate err in the manner mentioned above, the distortion deriving part 21c next derives the actual data 24b (the dashed-dotted line curve in
As mentioned above, in the vehicle-mounted apparatus 2 in the embodiment, the position identifying part 21a identifies the positions P1 and P2 of the mark images 71a and 72a in the captured image GC obtained by capturing the two marks 71 and 72 by the camera 5 (the step S21). Then, the error rate deriving part 21b derives the error rate err of the actual image height r′ to the design image height r, based on the positions P1 and P2 of the two mark images 71a and 72a (the step S22 to the step S30). Moreover, the distortion deriving part 21c derives the actual distortion characteristic (the actual data 24b) of the camera 5 based on the error rate err and the design characteristic (the design data 24a) (a step S31).
Since the actual distortion characteristic of the camera 5 can be derived from the positions P1 and P2 of the two mark images 71a and 72a in the captured image GC, in the manner mentioned above, large equipment, for example, many marks arranged in a grid in a work area is not required to derives the actual distortion characteristic. Therefore, the actual distortion characteristic of the camera 5 can be derived with relatively simple equipment.
Moreover, the error rate deriving part 21b sets the temporary value to the error rate err and derives the derived distance M between the two marks 71 and 72 based on the temporary value for the error rate err and on the positions P1 and P2 of the two mark images 71a and 72a in the captured image GC. Then, the error rate deriving part 21b derives the actual error rate err based on the evaluation value E that is the result of the comparison between the derived distance M and the actual distance m between the two marks 71 and 72. Therefore, the actual distortion characteristic of the camera 5 can be derived by a relatively simple algorithm.
Furthermore, the parameter obtaining part 21d obtains the installation parameters relating to the installation of the camera 5, based on the actual distortion characteristic of the camera 5 and on the captured image GC obtained by capturing the two marks 71 and 72 by the camera 5. Since the actual distortion characteristic of the camera 5 is used to obtain the installation parameters as mentioned above, accurate installation parameters can be obtained.
In addition, the image combiner 23 generates the combined image showing the surroundings of the vehicle 9 viewed from a virtual viewpoint, using the captured image captured by each of the plurality of cameras 5 and the actual distortion characteristic of each of the plurality of cameras 5. Therefore, a large problem where images of a same subject are misaligned can be prevented and a combined image correctly and entirely showing the surroundings of the vehicle 9 can be generated.
<7. Modification>
The embodiment of the invention has been explained above. However, the invention is not limited to the embodiment explained above but various modifications are possible. Examples of the modifications are explained below. All forms including the aforementioned embodiment and the modifications explained below may be combined arbitrarily.
In the aforementioned embodiment, the position identifying part 21a identifies the positions P1 and P2 of the mark images 71a and 72a in the captured image GC by the arithmetic calculation. However, the position identifying part 21a may identify the positions P1 and P2 of the mark images 71a and 72a in the captured image GC, based on positions specified by a user (worker). In this case, the user specifies the positions P1 and P2 of the mark images 71a and 72a by, for example, a cursor, viewing the captured image GC displayed on the display 26.
Moreover, in the aforementioned embodiment, the mark 70 is formed on the position marker 7 that can stand upright. However, the mark 70 may be formed on a principle surface of a position marker that cannot stand upright or may be formed on a floor surface of a work area.
Furthermore, in the aforementioned embodiment, it is explained that each of the plurality of cameras 5 captures an image of the two marks 70. However, each of the plurality of cameras 5 may capture an image of three or more marks 70. In other words, each of the plurality of cameras 5 should capture an image of at least two marks 70.
In addition, in the aforementioned embodiment, it is explained that the various functions are implemented by arithmetic processing performed by the CPU in accordance with a program. However, a part of the functions may be implemented by an electrical hardware circuit. On the other hand, a part of the functions implemented by a hardware circuit in the aforementioned embodiment may be implemented by software.
While the invention has been shown and described in detail, the foregoing description is in all aspects illustrative and not restrictive. It is therefore understood that numerous other modifications and variations can be devised without departing from the scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2012-119374 | May 2012 | JP | national |