Patent Application
20080036857
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the embodiments described below with reference to the drawings, video image positional relationship correction apparatuses according to the present invention are used for correcting the video image positional relationship between a video image at the back of a vehicle and a steering assist guide in a steering assist apparatus for the vehicle.
The controller 9 is provided at a position adjacent to the monitor 7 in front of the driver's seat of the vehicle 1. As shown in
The theoretical drawing circuit 8 functions as coordinate conversion means according to the present invention. The theoretical drawing means 8 and the controller 9 function as recognition means and correction means.
At this time, as shown in
A video image (mirror image) at the back of the vehicle captured by the CCD camera 2 and displayed on the screen of the monitor 7 is shown in
In performing the coordinate conversion between the road surface coordinate system and the monitor coordinate system, coordinate conversion parameters including attachment parameters for attaching the CCD camera 2 to the vehicle 1 and internal parameters of the CCD camera 2 itself are set.
Firstly, the following parameters are considered as the attachment parameters. When the CCD camera 2 is attached to the vehicle 1 in accordance with references, the CCD camera 2 is installed at a reference attachment position of a coordinate point (x, y, z) expressed by the road coordinate system at a reference attachment angle comprising a tilting angle ω, a direction angle γ, and a rotation angle θ. The tilting angle ω is an angle of downward inclination from the y-axis direction. The direction angle γ is an angle of inclination from a negative y-axis direction in a surface parallel to the xy surface. The rotation angle θ is an angle of attachment by rotating the CCD camera 2 about an optical axis of the lens 3.
However, in reality, the CCD camera 2 is attached to the vehicle with attachment errors with respect to the references. It is assumed that the CCD camera 2 is installed at an attachment position of a coordinate point (x+Δx, y+Δy, z+Δz) in the road surface coordinate system, at an attachment angle comprising a tilting angle ω+Δω, a direction angle γ+Δγ, and a rotation angle θ+Δθ. These parameters x+Δx, y+Δy, z+Δz, ω+Δω, γ+Δγ, θ+Δθ are the attachment parameters for attaching the CCD camera 2.
Further, the internal parameters may include a positional deviation amount ΔCx indicating deviation of the center of the CCD area sensor 4 in the positive x-axis direction with respect to the optical axis of the lens 3, a positional deviation amount ΔCy indicating deviation of the center of the CCD area sensor 4 in the positive y-axis direction with respect to the optical axis of the lens 3, a focus distance f+Δf of the CCD camera 2, and distortion constants Da, Db, Dc.
The distortion constants Da, Db, Dc are constants used in the following equation for defining a distortion coefficient D.
D=[(r−r0)/r0]×100=Da×r2+Db×r+Dc
where r0 is the image height which is determined without taking the distortion into account, r is the image height which is determined taking the distortion into account, and the image height is expressed by the distance from the intersection of the optical axis (extension line from the lens center) and the CCD area sensor surface to a target point on the CCD area sensor.
Further, in addition to these attachment parameters and internal parameters, the coordinate conversion parameters may include conversion constants to the screen of the monitor 7. The conversion constants are the X-axis magnification, the positional deviation in the X-axis direction, the Y-axis magnification, and the positional deviation in the Y-axis direction.
Of these parameters, for example, the following nine parameters are modified in the embodiment:
tilting angle ω+Δω, direction angle γ+Δγ, rotation angle θ+Δθ, distortion constants Da and Db, X-axis magnification, positional deviation in the X-axis direction, Y-axis magnification, and positional deviation in the Y-axis direction.
It is difficult to directly measure these parameters, and calculate these parameters based on other parameters.
Next, operation of the video image positional relationship correction apparatus according to this embodiment will be described.
Firstly, the actual video image including reference points P1 to P6 as actual targets is captured by the CCD area sensor 4 through the lens 3. A signal representing the actual image captured by the CCD area sensor 4 is transmitted to the signal processing IC 5, and outputted to the superimposing circuit 6. Further, signals representing virtual target points R1 to R6 are inputted from the theoretical drawing circuit 8 to the superimposing circuit 6.
At this time, derivation of the virtual target points R1 to R6 in the theoretical drawing circuit 8 will be described. The reference positions P1 to P6 on the road surface are determined in advance, and the stop position of the vehicle 1 with respect to these reference points P1 to P6 is also determined in advance. Therefore, the respective virtual target points R1 to R6 are derived theoretically based on the coordinate conversion parameters before modification which are determined without taking the errors of the coordinates of the reference points P1 to P6 in the road surface coordinate system into account. The theoretical drawing circuit 8 outputs the coordinates determined theoretically in this manner as coordinate data of the virtual target points R1 to R6 in the monitor coordinate system to the superimposing circuit 6.
In the superimposing circuit 6, based on the signal representing the actual image and the coordinate data representing the virtual target points R1 to R6 outputted from the theoretical drawing circuit 8, the actual image and the virtual target points R1 to R6 drawn by dotted lines are superimposed on the screen of the monitor 7. At this time, if the attachment parameters and the internal parameters of the CCD camera 2 and the conversion constants to the screen of the monitor 7 are ideal, the positions of the video image reference points Q1 to Q6 on the monitor screen indicating the actually captured video image reference points P1 to P6 and the positions of the virtual target points R1 to R6 are overlapped with each other on the screen of the monitor 7. However, for example, if the CCD camera 2 is attached with attachment errors with respect to the references, or if the optical axis of the lens 3 of the CCD camera 2 is not in alignment with the center of the CCD area sensor 4, as shown in
In this case, the driver manipulates the direction buttons 10 of the controller 9 such that the virtual target point R1 is overlapped on the target reference point Q1 initially. The movement amount of the virtual target point R1 inputted by the direction buttons 10 is inputted to the theoretical drawing circuit 8. Then, if the driver presses the decision button 11 when the virtual target point R1 is overlapped on the video image reference point Q1, the signal of the decision button 11 is inputted to the theoretical drawing circuit 8. Thus, the theoretical drawing circuit 8 recognizes the coordinate of the video image reference point Q1 in the monitor coordinate system. By repeating the manipulation to move the virtual target points R2 to R6 successively, the theoretical drawing circuit 8 recognizes the coordinates of the video image reference points Q2 to Q6 in the monitor coordinate system.
Next, when the calculation button 12 of the controller 9 is pressed, by a calculation method as described later, the theoretical drawing circuit 8 calculates the coordinate conversion parameters after modification to take the errors into account such that the virtual target points R1 to R6 substantially match the video image reference points Q1 to Q6. For example, at this time, coordinates of new virtual target points R1 to R6 and lines extending between those coordinates are calculated based on the coordinate conversion parameters after modification, and the video image is displayed again on the screen of the monitor 7 by the superimposing circuit 6. Thus, the driver can confirm whether the correction is properly carried out or not based on the positional relationship with the reference points P1 to P6.
After the correction is finished in this manner, the theoretical drawing circuit 8 produces data of the virtual video image in the monitor coordinate system, e.g., display data of the steering assist guide based on the coordinate conversion parameters after modification.
The theoretical drawing circuit 8 calculates the coordinates of the virtual target points R1 to R6 to be displayed on the screen of the monitor 7 using the coordinate conversion parameters before modification. Then, the theoretical drawing circuit 8 determines the coordinate conversion parameters based on the virtual target points R1 to R6, the video image reference points Q1 to Q6, and the coordinate conversion parameters before modification. Next, a method of carrying out those processes by the theoretical drawing circuit 8 will be described.
The coordinate values Xqm and Yqm of the video image reference points Qm (m=1 to 6) in the monitor coordinate system are expressed by the following equations using functions F and G, based on the coordinate values Xpm, Ypm, Zpm of the reference points Pm (m=1 to 6) in the road surface coordinate system and the above-mentioned nine coordinate conversion parameters Kn (n=1 to 9) which require modification, and the other parameters Kj (j=10 to 16) which do not require modification.
Xqm=F(Xpm, Ypm, Zpm, Kn, Kj)+DXm
Yqm=G(Xpm, Ypm, Zpm, Kn, Kj)+DYm
DXm and DYm are deviations between the X coordinates and the Y coordinates of the virtual target points calculated using the functions F and G, and the coordinate values Xqm and Yqm of the video image reference points Qm. If the reference points Pm are drawn on the road surface, Zpm=0.
That is, by expressing the X coordinates and Y coordinates of the six video image reference points Qm, twelve relational expressions are generated in total for nine coordinate conversion parameters Kn.
At this time, the coordinate conversion parameters Kn are determined such that the square-sum of the deviations DXm and Dym expressed by the following equation is the minimum.
S=Σ(DXm2+DYm2)
That is, an optimization problem for minimizing S is solved. Known optimizing methods such as a simplex method, a steepest descent method, a Newton method, and a quasi Newton method are used to solve the problem. Values of the coordinate conversion parameters before modification are used as initial values of the coordinate conversion parameters Kn at the time of repeated calculations.
In this manner, the coordinate conversion parameters Kn are determined, and the virtual video image data in the monitor coordinate system, e.g., the display data of the steering assist guide is produced again based on the coordinate conversion parameters after modification by the theoretical drawing circuit 8 to properly correct the positional relationship between the actual video image and the virtual video image.
Thus, even if the optical axis of the lens 3 is not in alignment with the center of the CCD area sensor 4, and the CCD camera 2 is not properly attached to the vehicle 1 in accordance with the references, the positional relationship between the actual video image and the virtual video image is corrected properly without physically adjusting the optical axis of the lens 3, and without any adjustment operation for attaching the CCD camera 2 in accordance with the references to the vehicle 1 with high accuracy. That is, the positional relationship between the image at the back of the vehicle as the actual video image and the steering assist guide as the virtual image is corrected properly.
Since the coordinate conversion parameters are calculated using the relational expressions, and the number of the relational expressions is larger than the number of the coordinate conversion parameters, therefore, even if errors occur at the time of recognizing the deviation of the coordinate between the virtual target point and the video image reference point by manipulation of the controller 9, the coordinate conversion parameters which do not require modification include errors, or errors occur due to parameters other than the coordinate conversion parameters enumerated above, it is possible to obtain the appropriate coordinate conversion parameters, and carry out the correction accurately.
In the first embodiment, the twelve relational expressions are produced using the six video image reference points Qm for the nine coordinate conversion parameter Kn. However, the present invention is not limited in this respect. As long as the number of the relational expressions is larger than the number of the coordinate conversion parameters to be calculated, other configurations can be envisaged. For example, ten relational expressions may be produced using five video image reference points Qm, or a larger number of relational expressions may be produced using seven or more video image reference points Qm.
Further, the number of the coordinate conversion parameters is not limited to nine, and can be determined freely.
In the first embodiment, the superimposing circuit 6 superimposes the image signal of the actual video image outputted from the signal processing IC 5 of the CCD camera 2 and the signal of the virtual image outputted from the theoretical drawing circuit 8 on the monitor 7. In the second embodiment, the image signal of the actual video image outputted from the image processing IC 5 of the CCD camera 2 is converted by the A/D converter circuit 15 into image data, and the image data is temporarily stored in the image memory 16. Data of the virtual image outputted from the theoretical drawing circuit 8 is added to the image data of the actual image on the image memory 16. Then, the image data after addition of the virtual image data is transmitted to the monitor 7 through the D/A converter circuit 17, and the actual video image and the virtual image are superimposed on the screen of the monitor 7.
As described above, the video image positional relationship correction apparatus according to the present invention is also applicable to the video image display device in which the image data is temporarily stored in the image memory 16.
In the first to third embodiments, the reference points P1 to P6 are drawn as the actual targets on the road surface. Alternatively as shown in
Further, part of the vehicle 1 in the area captured by the CCD camera 2 may be regarded as the actual target.
The present invention is not limited to the above-mentioned embodiments. The following modifications may be made to the embodiments in carrying out the present invention.
The shape of the reference points as the actual target is not limited to have a circular shape. The reference points may have various shapes.
In the first and second embodiments, the controller 9 is used for manipulation by the driver. Alternatively, instead of using the controller, the monitor 7 may include a touch panel equipped with direction buttons, a decision button, and a calculation button or a jog switch or the like. Any means which is manipulated to overlap the virtual target point and the video image reference point can be used. Further, the order of manipulation is not limited to the above-mentioned embodiments. Various orders can be adopted without deviating the scope of the invention.
Further, the manipulation is not limited to the operation of overlapping the virtual target point on the image reference point. Any manipulation can be adopted as long as it makes it possible to recognize which virtual target point of the virtual target points R1 to R6 corresponds to the coordinate of the image reference point on the screen. For example, the direction buttons or the like may be used to recognize the coordinate of the image reference point, and then, recognize which virtual target point corresponds to the image reference point. The means used to recognize which virtual target point corresponds to the image reference point may be manipulation of the driver. When the virtual target point is not deviated from the image reference point significantly, the nearest image reference point may be recognized automatically as the image reference point corresponding to the virtual target point. In the case of the automatic recognition, the virtual target point may not be displayed.
The coordinate conversion parameters which require modification are not limited to the parameters used in the above-mentioned embodiments. Any parameters may be adopted as long as at least the internal parameters of the camera are included.
In the first through fourth embodiments, the virtual video image produced by the theoretical drawing circuit 8 is corrected. However, the present invention is not limited in this respect. Alternatively, the actual video image captured by the CCD camera 2 may be corrected.
Further, in the embodiments, in particular, the video image of the steering assist apparatus at the back of the vehicle is corrected. However, the present invention is not limited in this respect. The video image positional relationship correction apparatus of the present invention is also applicable to video image correction in other apparatuses that superimpose the actual image and the virtual image.
According to the present invention, even if deviation occurs in the optical axis of the lens, the positional relationship between the actual video image and the virtual video image is corrected properly. Further, attachment of the optical axis and the CCD camera is not adjusted physically but corrected by software. Therefore, the video image positional relationship is corrected with high accuracy at low cost. Further, the accuracy of displaying the two-dimensional distance and the guideline is improved. Therefore, the present invention is applicable to the measurement related field other than the field of the vehicle.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2004-023673 | Jan 2004 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP04/16454 | 10/29/2004 | WO | 00 | 4/13/2007 |
