Moving object locating device, moving object locating method, and computer product

Abstract

Camera installation flexibility on a moving object can be increased and accuracy in identifying the moving object improved by identifying a moving object origin trajectory as a potential position at which the moving object origin exists under the following conditions: 1) A distance between the moving object origin position and a reference virtual camera is constant.2) A direction of virtual camera VCij as viewed from the moving object coordinate system (moving object origin Om) is constant (angle formed by two lines that direct to marker Pj and to the moving object origin position Cij as viewed from the virtual camera VCij is constant.)3) Virtual camera VCij exists on marker circle (angle between markers Pi and Pj as viewed from virtual camera VCij is constant).

Description

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a locating method of a moving object according to an embodiment of this invention;

FIG. 2 is a diagram showing a moving object origin trajectory according to an embodiment of this invention;

FIG. 3 is a diagram showing various coordinate systems according to an embodiment of this invention;

FIG. 4 is a diagram showing how an azimuthal angle of a marker is estimated upon a marker being photographed by an arbitrary camera;

FIG. 5 is a diagram showing how an inter-marker angle is estimated;

FIG. 6 is a diagram showing how a moving object origin from an inter-marker angle in global coordinate system is estimated;

FIG. 7 is a diagram showing a method to calculate an initial value of a variable from a measured GPS value;

FIG. 8 is a diagram showing a solution space of an evaluation function;

FIG. 9 is a block diagram showing a moving object locating system according to an embodiment of this invention;

FIG. 10 is a diagram showing content of marker DB;

FIG. 11 is a block diagram showing a moving object locating system according to an embodiment of this invention;

FIG. 12 is a block diagram showing an identifying unit;

FIG. 13 is a flowchart showing a moving object-locating procedure according to an embodiment of this invention;

FIG. 14 is a flowchart showing a procedure of a variable decision process;

FIG. 15 is a diagram showing neighboring points around a focused point;

FIG. 16 is a diagram showing a locating method based on markers photographed by cameras installed on a moving object;

FIG. 17 is a diagram showing an example of an all-direction camera image and a relevant optical system; and

FIG. 18 is a diagram-showing an installation example for a plurality of cameras on a vehicle.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Exemplary embodiments according to the present invention will be explained in detail below with reference to accompanying drawings.

The example herein employs a plurality of conventional cameras to detect the markers in lieu of an all-direction camera. For vehicles equipped with a plurality of cameras at the front, sides, rear, etc., for obtaining a peripheral view relevant to the vehicle, these existing cameras may be utilized.

FIG. 1 is a diagram showing a principle of moving object-locating according to an embodiment of this invention. The moving object-locating assumes a virtual camera VCij (i.e., an optical center of camera) at a position from where cameras CMi and CMj (i, j=1, 2, 3, i≠j) mounted on a moving object M can simultaneously photograph two markers P_i and P_j. This camera is herein called “virtual camera VC_ij”, and an installation position of this camera is herein expressed as V_ij.

To distinguish a camera CMi mounted on a moving object M from the virtual camera VCij, the former may be called as a “Real camera CMi”. Here, “Marker” means a photograph subject of the camera CMi, for example, a traffic signal, a road traffic sign, a structural body such as a utility pole, or an item that is attached to such a body and is recognizable from the surrounding area.

For the following descriptions and equations, an operation principle is explained using a pair of markers P_i and P_j that are photographed by cameras CMi and CMj and as the principle is identical, explanation is omitted regarding a pair of markers P_j and P_k that are photographed by cameras CMj and CMk.

As shown in FIG. 1, the virtual camera VCij is virtually located at a position from which two markers P_i and P_j that are actually photographed by different cameras CMi and CMj mounted on the moving object M may be photographed simultaneously. That is, the virtual camera VCij is virtually located at an intersecting point of a straight line that passes through the camera CMi and the marker P_i and a straight line that passes through the camera CMj and the marker P_j.

And, a virtual camera VCjk is virtually located at a position from which two markers P_j and P_k that are actually photographed by different cameras CMj and CMk mounted on the moving object M. That is, the virtual camera VCjk is virtually installed at an intersecting point of a straight line that passes through the camera CMi and the marker P_j and a straight line that passes through the camera CMk and the marker P_k.

In this way, the virtual camera VCij is located at a position where the inter-marker angle θ_ij can be calculated and assuming a moving object origin O_m to be the optical center, the position V_ij of virtual camera VCij on a moving object coordinate system can be calculated from the direction of cameras CMi and CMj and markers P_i and P_j. Similarly, the virtual camera VCjk is set up at a position where the inter-marker angle θ_jk can be calculated and assuming the moving object origin O_m to be the optical center, the position V_jk of virtual camera VCjk on a moving object coordinate system can be calculated from the direction of cameras CMj and CMk and markers P_j and P_k.

In the described related art, a circle that passes through the marker markers Pi and Pj. and the moving object origin Om was called the marker circle Eij, however in this embodiment, a circle with a center position Oij and a radius rij that passes through the markers Pi and Pj and the installation position Vij of virtual camera VCij is called the marker circle Eij. The marker circle can identify a circle with its center position Oij and radius rij (shown in FIG. 1, Eij and Ejk) by substituting the inter-marker angle θ_ij having a base at the virtual camera VCij into equations 1.1 and 1.2.

From the relative position relationship between the installation position V_ij of virtual camera VCij and the moving object origin position, a moving object origin trajectory T that becomes a position candidate where the moving object origin position C_ij exists is identified. FIG. 2A is a diagram showing the moving object origin trajectory T according to an embodiment of this invention. The moving object origin trajectory T is an aggregate of position coordinates that satisfy conditions 1 to 3 following.

condition 1: a distance d between the moving object origin position C_ij and the virtual camera VCij is constant.

condition 2: a direction of virtual camera VCij as viewed from the moving object coordinate system (moving object origin O_m) is constant (angle formed by two lines that direct to marker P_j and to the moving object origin position C_ij as viewed from the virtual camera VCij is constant.)

condition 3 the virtual camera VCij exists on the marker circle E_ij (or, angle formed by two lines that direct to the marker P_i and P_j starting from the virtual camera position VCij is constant.)

FIG. 2B is a diagram showing the moving object origin trajectory T that satisfies the described conditions 1 to 3. In case that there is a plurality of moving object origin trajectories T (for example, in case that more than 3 marker points P₁, P₂, P₃, . . . were observed.), the position of moving object M can be identified by calculating an intersecting point coordinate of each moving object origin trajectory T. The intersecting point coordinate of each moving object origin trajectory T can be calculated by a numerical solution method such as a successive approximation method.

FIG. 3 is a diagram showing the coordinate systems according to an embodiment of this invention. In FIG. 3, a moving object coordinate system 301 is a coordinate system that expresses a plane that is parallel to the plane in which the moving object M is traveling (hereinafter, such an orientation is described as “horizontal”) and is defined by a Y-axis corresponding to direction travel by the moving object M and an X-axis that is perpendicular to the Y-axis.

A camera coordinate system 302 is a coordinate system that assumes a camera origin O_c to be the optical center of each camera CMi and expresses a horizontal plane and is defined by a Y_ci-axis corresponding to an optical axis of camera CMi and an X_ci-axis that is perpendicular to Y_ci-axis. The camera coordinate system 302 exists for each camera CMi. A global coordinate system 303 is a coordinate system that expresses a horizontal plane and is defined by a Y_w-axis corresponding to a north/south direction and an X_w-axis corresponding to an east/west direction, assuming an origin O_w to be a predetermined reference point. A Z-axis (not shown in FIG. 3) corresponding to a vertical direction in each coordinate system is an axis that passes vertically through the horizontal plane. It is assumed that there is no change of scale among coordinate systems 301 to 303 and conversion of coordinate system is made possible by executing rotational and parallel movement around the Z-axis. Reference characters T_cm and T_mw in FIG. 3 indicate a traveling distance between coordinates.

FIG. 4 is a diagram showing how marker direction is estimated upon a certain marker P_i being photographed by an arbitrary camera CMi. Assuming that the marker P_i was photographed at a position (x_p, y_p) in an image coordinate system 400 (X_p-Y_p) photographed by the camera CMi, the marker P_i exists in a direction that was rotated by an angle α from an optical axis direction (Y_c-axis) in the camera coordinate system 302. At this time, α is expressed by equation 2.

$\begin{array}{cc}\alpha =\arctan \frac{2x_p\tan \frac{{\Theta }_H}{2}}{W_p}& \left(2\right)\end{array}$

Where, Θ_H is a horizontal image angle of camera CMi and W_p is a width of an image. When conversion from the camera coordinate system 302 to the moving object coordinate system 301 is executed by horizontal movement of (t_x, t_y) and rotational movement by an angle β, the camera origin (optical center) O_ci of the camera CMi and a straight line L_i (marker line) passing through the marker P_i are expressed by equation 3.

$\begin{array}{cc}\frac{x-t_x}{-\sin \left(\beta -\alpha \right)}=\frac{y-t_y}{\cos \left(\beta -\alpha \right)}& \left(3\right)\end{array}$

Similarly, for the other camera and marker, its camera origin (optical center) and straight line (marker line) passing through the marker can also be calculated by equation 3.

FIG. 5 is a diagram showing how the inter-marker angle is estimated. As shown in FIG. 5, the inter-marker angle θ_ij is estimated by using the marker lines L_i and L_j obtained from a plurality of cameras (CMi, CMj). More concretely, the intersecting point of each marker line L_i and L_j is first calculated, then the angle (inter-marker angle θ_ij) formed by P_i and P_j taking this intersecting point as a reference is calculated. In case that two arbitrary markers P_i and P_j were photographed by the same camera CMi, the intersecting point of marker lines L_i and L_j coincides with the camera origin (optical center) O_ci of the camera CMi.

In case that P_i and P_j were photographed by different cameras CMi and CMj, point Q_ij becomes an intersecting point. The intersecting point Q_ij is called “virtual optical center Q_ij”. This virtual optical center Q_ij is expressed by equations 4.1 to 4.5.

$\begin{array}{cc}{Q}_{\mathrm{ij}}=\left(\frac{A_jB_it_{\mathrm{ix}}-A_iB_jt_{\mathrm{jx}}+A_iA_j\left(t_{\mathrm{jy}}-t_{\mathrm{iy}}\right)}{A_jB_i-A_iB_j},\frac{A_jB_it_{\mathrm{jy}}-A_iB_jt_{\mathrm{iy}}+B_iB_j\left(t_{\mathrm{ix}}-t_{\mathrm{jx}}\right)}{A_jB_i-A_iB_j}\right)& \left(4.1\right)\\ A_i=-\sin \left({\beta }_i-{\alpha }_i\right)& \left(4.2\right)\\ B_i=\cos \left({\beta }_i-{\alpha }_i\right)& \left(4.3\right)\\ A_j=-\sin \left({\beta }_j-{\alpha }_j\right)& \left(4.4\right)\\ B_j=\cos \left({\beta }_j-{\alpha }_j\right)& \left(4.5\right)\end{array}$

The inter-marker angle θ_ij* of two markers P_i and P_j as viewed from the virtual optical center Q_ij is expressed by equation 5.

θ*_ij=α_i−β_i−α_j+β_j  (5)

An angle φ_im formed by one marker P_i as viewed from the virtual optical center Q_ij and the moving object origin O_m is expressed by equations 6.1 to 6.3, and an angle φ_im formed by another marker P_j as viewed from the virtual optical center Q_ij and the moving object origin O_m is expressed by equations 6.2, 6.4, and 6.5.

$\begin{array}{cc}{\phi }_{\mathrm{im}}=\arcsin \overrightarrow{k_i}\times \overrightarrow{v_{\mathrm{ij}}}& \left(6.1\right)\\ v_{\mathrm{ij}}=\frac{-Q_{\mathrm{ij}}}{Q_{\mathrm{ij}}}& \left(6.2\right)\\ \overrightarrow{k_i}=\left(-\sin \left({\beta }_i-{\alpha }_i\right),\cos \left({\beta }_i-{\alpha }_i\right)\right)=\left(A_i,B_i\right)& \left(6.3\right)\\ {\phi }_{\mathrm{jm}}=\arcsin \overrightarrow{k_j}\times \overrightarrow{v_{\mathrm{ij}}}& \left(6.4\right)\\ \overrightarrow{k_j}=\left(-\sin \left({\beta }_j-{\alpha }_j\right),\cos \left({\beta }_j-{\alpha }_j\right)\right)=\left(A_j,B_j\right)& \left(6.5\right)\end{array}$

where, k_i is a unit line indicating direction from the virtual optical center Q_ij to marker P_i, and k_j is a unit line indicating direction from the virtual optical center Q_ij to marker P_j.

As shown in FIG. 5, a camera origin Ocj is an origin (optical center) of the camera CMj, the Yci-axis is a coordinate axis of camera coordinate system corresponding to optical axis direction of the camera CMj, the Xcj-axis is a coordinate axis of camera coordinate system of the camera CMj expressing a horizontal plane by perpendicularly crossing the Ycj-axis.

FIG. 6 is a diagram showing how the moving object origin Om is estimated in the global coordinate system 303 from the inter-marker angle θ_ij*. Hereafter, each symbol having a “*” mark means that it is a value (coordinate position or degree) in the global coordinate system. For example, marker P_i* indicates that this is a coordinate position of the marker P_i in the global coordinate system 303. As shown in FIG. 6, the position of the virtual optical center Q_ij estimated from the inter-marker degree θ_ij* in the global coordinate system 303 is given by a point Q_ij* on a circle having the center position O_ij* and the radius r_ij*. The center position Q_ij* and the radius r_ij* are expressed by equations 7.1 and 7.2.

$\begin{array}{cc}{O}_{\mathrm{ij}}^{*}=\left(\frac{p_{\mathrm{ix}}^{*}+p_{\mathrm{jx}}^{*}}{2},\frac{p_{\mathrm{iy}}^{*}+p_{\mathrm{jy}}^{*}}{2}\right)-\frac{1}{\tan {\theta }_{\mathrm{ij}}^{*}}\left(\frac{p_{\mathrm{jy}}^{*}-p_{\mathrm{iy}}^{*}}{2},\frac{p_{\mathrm{jx}}^{*}-p_{\mathrm{ix}}^{*}}{2}\right)& \left(7.1\right)\\ r_{\mathrm{ij}}^{*}=\frac{1}{2\sin {\theta }_{\mathrm{ij}}^{*}}\sqrt{{\left(p_{\mathrm{jx}}^{*}-p_{\mathrm{ix}}^{*}\right)}^2+{\left(p_{\mathrm{jy}}^{*}-p_{\mathrm{iy}}^{*}\right)}^2}& \left(7.2\right)\end{array}$

Where, P_i*=(p_ix*, p_iy*), P_j*=(p_jx*, p_jy*) are coordinates of marker positions in the global coordinate system 303.

Since t distance |Q_ij| between the moving object origin O_m and the virtual optical center O_ij*, and the angle θ_im formed by moving object origin O_m and the marker P_i are common in both the moving object coordinate system 301 and the global coordinate system 303, the position C_ij* of the moving object M satisfies equation 8.

$\begin{array}{cc}{C}_{\mathrm{ij}}^{*}=Q_{\mathrm{ij}}^{*}+\frac{Q_{\mathrm{ij}}}{P_i^{*}-Q_{\mathrm{ij}}^{*}}\left[\begin{array}{cc}\cos {\phi }_{\mathrm{im}}& -\sin {\phi }_{\mathrm{im}}\\ \sin {\phi }_{\mathrm{im}}& \cos {\phi }_{\mathrm{im}}\end{array}\right]\left(P_i^{*}-Q_{\mathrm{ij}}^{*}\right)& \left(8\right)\end{array}$

Where, the position C_ij* of the moving object M is merely known to exist on a circle expressed by equations 7.1 and 7.2, and therefore can not be determined by only equation 8. If the virtual optical center Q_ij* is expressed using a variable μ_ij in a polar coordinate system, it can be re-written as equation 9.

Q* _ij(μ_ij)=O*_ij+r*_ij(cos μ_ij, sin μ_ij)  (9)

The position C_ij* of the moving object M is expressed as a function of the variable μ_ij by equation 10.

$\begin{array}{cc}{C}_{\mathrm{ij}}^{*}\left({\mu }_{\mathrm{ij}}\right)=Q_{\mathrm{ij}}^{*}\left({\mu }_{\mathrm{ij}}\right)+\frac{Q_{\mathrm{ij}}}{P_i^{*}-Q_{\mathrm{ij}}^{*}\left({\mu }_{\mathrm{ij}}\right)}\left[\begin{array}{cc}\cos {\phi }_{\mathrm{im}}& -\sin {\phi }_{\mathrm{im}}\\ \sin {\phi }_{\mathrm{im}}& \cos {\phi }_{\mathrm{im}}\end{array}\right]\left(P_i^{*}-Q_{\mathrm{ij}}^{*}\left({\mu }_{\mathrm{ij}}\right)\right)& \left(10\right)\end{array}$

Where, the position C_ij* can be calculated for an arbitrary pair of markers P_i and P_j. For example, if the position C_jk* is calculated for another pair of markers P_j and P_k by equation 10, the position C_jk* is identical to the position C_ij*. Therefore, by calculating the variables μ_ij and μ_jk that make a value of F(μ_ij, μ_jk) equal to 0 in equation 11, and by substituting the variables μ_ij and μ_jk into equation 10, it is possible to determine the moving object position C_ij* in the global coordinate system 303.

F(μ_ij,μ_jk)=|C*_ij(μ_ij)−C*_jk(μ_jk)|  (11)

It is also possible to determine the azimuthal angle ω(μ_ij) with which the direction of the Ym-axis of the moving object coordinate system 301 indicates the direction of moving object M by equations 12.1 and 12.2.

$\begin{array}{cc}\omega \left({\mu }_{\mathrm{ij}}\right)=\arcsin \overrightarrow{k_i}\times \overrightarrow{k_i^{*}}& \left(12.1\right)\\ \stackrel{\_}{k_i^{*}}=\frac{P_i^{*}-Q_{\mathrm{ij}}^{*}\left({\mu }_{\mathrm{ij}}\right)}{P_i^{*}-Q_{\mathrm{ij}}^{*}\left({\mu }_{\mathrm{ij}}\right)}& \left(12.2\right)\end{array}$

It is also possible to calculate the variables μ_ij and μ_jk by using an analytical method or by using a numerical analysis method. When using a numerical analysis method, although the result depends on an initial value, it is possible to obtain a good initial value by using a previous position as an initial value for the moving object position in the case of continuous measurement or by using a value that was obtained from another moving object locating system. In case of the latter method, accuracy of other moving object locating system may be relatively low. For example, moving object locating systems such as a conventional GPS or an electronic compass may also be utilized.

Here, a calculation method for the initial value of variable μ_ij from a measured GPS value is explained. FIG. 7 is a diagram showing the calculation method of initial value of variable μ_ij from the measured GPS value. As shown in FIG. 7, a position that was obtained by converting the latitude and longitude measured by GPS to the ones in the global coordinate system 303 (orthogonal coordinate system) is indicated as G_ij*=(g_ijx, g_ijy) and the virtual optical center coordinate is indicated by R_ij*.

At this time, although the coordinate value of R_ij* is not known, it is known that its position is on a circle E_ij that is expressed by a center position O_ij* and radius r_ij* obtained from equations 7.1 and 7.2. Assuming that an angle formed by two lines directed from the marker P_i* to the virtual optical center coordinate and a line directed from R_ij* to the latitude/longitude coordinate G_ij* can be approximated by the angle φ_im (refer to equations 6.1 to 6.3), the virtual optical center coordinate R_ij* exists on the circle E expressed by equations 13.1 and 13.2.

$\begin{array}{cc}\sum _{\mathrm{ij}}=\left(\frac{p_{\mathrm{ix}}+g_{\mathrm{ijx}}}{2},\frac{p_{\mathrm{iy}}+g_{\mathrm{ijy}}}{2}\right)-\frac{1}{\tan {\phi }_{\mathrm{im}}}\left(\frac{g_{\mathrm{ijy}}-p_{\mathrm{iy}}}{2},\frac{g_{\mathrm{ijx}}-p_{\mathrm{ix}}}{2}\right)& \left(13.1\right)\\ {\sigma }_{\mathrm{ij}}=\frac{1}{2\sin {\phi }_{\mathrm{im}}}\sqrt{{\left(g_{\mathrm{ijx}}-P_{\mathrm{ix}}\right)}^2+{\left(g_{\mathrm{ijy}}-p_{\mathrm{iy}}\right)}^2}& \left(13.2\right)\end{array}$

Thus, the virtual optical center coordinate R_ij* can be obtained by calculating the intersecting point of the circle E and the circle E_ij. The virtual optical center coordinate R_ij* can be expressed by equation 14.

R* _ij =O* _ij +r* _ij(cos λ_ij, sin λ_ij)  (14)

In equation 14, a value that best satisfies the variable λ_ij can be determined as an initial value of the variable μ_ij.

Next, the numerical analysis method for variable μ_ij and μ_jk is described concretely. FIG. 8 is a showing a curved surface as a solution space of the evaluation function F(μ_ij, μ_jk) expressed by equation 11. Both of the variables μ_ij and μ_jk have a definition region of [0, 2π] and F(μ_ij, μ_jk) has a unique value equal to or greater than 0 for each variable μ_ij and μ_jk. By obtaining the values (μ_ij, μ_jk) that make F(μ_ij, μ_jk) equal to 0 on this curved surface S, it is possible to identify the moving object coordinate origin O_m.

The values (μ_ij, μ_jk) that make F(μ_ij, μ_jk) equal to 0 can be obtained by using a numerical analysis method. Assuming that an arbitrary position (λ_ij,λ_jk) is an initial value of (μ_ij, μ_jk), a position where F becomes 0 (or minimum value) when moving the position (λ_ij, λ_jk) in direction in which the value decreases in the solution space of F(μ_ij, μ_jk) is a desirable value for (λ_ij, μ_jk).

However, since the curved surface of F(μ_ij, μ_jk) is not necessarily decreasing monotonically and the solution does not necessarily converge to a correct solution, how the initial value is selected is important. Hence, conversion to an incorrect solution is avoided either by using the (μ_ij, μ_jk) value of a previous moving object-locating or by using the initial value of the variable ij obtained from a rough position calculation result by the GPS.

FIG. 9 is a diagram showing a configuration of a moving object locating system according to an embodiment of this invention. As shown in FIG. 9, the moving object locating system includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, a random access memory (RAM) 903, a hard disk drive (HDD) 904, a hard disk (HD) 905, an optical disk drive 906, an optical disk 907 as an example of a removable recording medium, a display 908, an interface (I/F) 909, an input key 910, a microphone 911, a speaker 912, a GPS 913, an electronic compass 914, and a plurality of cameras CM. Each component is connected by a bus 900.

The CPU 901 governs operation control of the entire moving object locating system. The ROM 902 stores a program such as a boot up program. The RAM 903 is used as a work area of the CPU 901. The HDD 904 controls reading/writing of data in the HD 905 under the control of the CPU 901. The HD 905 stores data written under the control of the HDD 904.

The optical disk drive 906 controls reading/writing of the optical disk 907 under the control of the CPU 901. The optical disk 907 stores data written under the control of the optical disk drive 906 and allows the moving object locating system to read the stored data in the optical disk 907.

The display 908 displays data such as a document, an image, function information, as well as a cursor, an icon, toolbox, etc. For the display 908, a cathode ray tube (CRT), thin film transistor (TFT) liquid crystal display, plasma display, etc. for example, may be adopted.

The I/F 909 is connected to a network such as the internet through a communication line, and is also connected to external equipment via the network. Furthermore, the I/F 909 acts as an interface between the network and the internal moving object locating system and controls input/output of data from/to external equipment. For the I/F 909, a modem, a local area network (LAN) adapter, etc. for example, may be adopted.

The input key 910 is a set of buttons for inputting characters, numerical values, or various commands and serves as a data entry device. It may be a touch-panel. The microphone 911 receives an audio signal from an external source and converts it to a digital signal. The speaker 912 outputs sound according to the audio signal converted from the digital signal.

The GPS 913 includes an antenna for receiving an electromagnetic wave from a GPS satellite, a tuner for demodulating the received signal, and a logic circuit for calculating a current position according to the demodulated information. Furthermore, the current position of the moving object M can be determined by receiving an electromagnetic wave from the GPS satellite and calculating a geometrical position against the GPS satellite. The electronic compass 914 is an integrated circuit (IC) for calculating the direction of the moving object and includes an earth magnetism sensor. It detects the earth magnetism from north to south generated by the earth using the earth magnetism sensor and calculates the azimuthal angle to the moving object. The plurality of cameras CM is an aggregate of multiple cameras CM1 to CM_n looked at through by the moving object M.

FIG. 10 is a diagram showing content stored in a marker database DB 1000. As shown in FIG. 10, the marker DB 1000 stores the marker data for each marker P_i including a marker ID, a marker type, a marker latitude, a marker longitude, a latitudinal distance, and a longitudinal distance.

The marker type is information to identify the type of the marker P_i. For example, “0” is defined as a reference point, “1” is defined as a traffic signal, “2” is defined as a road traffic sign, “3” is defined as a utility pole, etc. The marker latitude means the latitude of the marker P_i, and the marker longitude means the longitude of the marker P_i.

Longitudinal distance means the distance in the longitudinal direction of P_i from the reference point, and latitudinal distance means the distance in the latitudinal direction of P_i from the reference point. Specifically, the marker DB 1000 realizes its function by recording media such as the ROM 902, the RAM 903, the HD 905, and the optical disk 907, for example, shown in FIG. 9.

FIG. 11 is a block diagram of the moving object locating system according to an embodiment. As shown in FIG. 11, a moving object locating system 1100 includes the marker DB 1000, the plurality of cameras CM ( . . . , CMi, CMj, CMk, . . . ), an image processing unit 1101 ( . . . , 1101i, 1101j, 1101k, . . . ), a detecting unit 1102, an identifying unit 1103 and a locating unit 1104.

Each camera CMi, CMj, and CMk are set so as to photograph the horizontal road environment from the moving object M. The installation position coordinates (t_ix, t_iy) (t_jx, t_jy) and (t_kx, t_ky) of each camera CMi, CMj, and CMk in the camera coordinate system 302 and the rotation angles β_i, β_j, and β_k that express the difference of coordinate axis from that of the moving object coordinate system 301 have already been determined. The installation position coordinates (t_ix, t_iy) (t_jx, t_jy) and (t_kx, t_ky) and the rotation angles β_i, β_j, and β_k of each camera CMi, CMj, and CMk are read into the detecting unit 1102 herein later described.

The image processing unit 1101 extracts images of the markers P_i, P_j and P_k from images photographed outside the moving object M by the plurality of cameras CM. Image extraction and recognition of the markers P_i, P_j and P_k in the photographed images may be executed by using an appropriate conventional technology. In general, upon the markers P_i, P_j and P_k being traffic signals, the markers can be extracted by identifying the color (such as red, yellow, and green) and shape (such as a circle) of a light that a traffic signal lens might emit. Although the image processing unit 1101 is built as 1 to 1 with a camera, it is also possible to take such a structure that processes the group of camera CM with a single image processing unit 1101.

The detecting unit 1102 detects position and direction of the markers P_i, P_j and P_k that are contained in each image photographed of the outside environment of the moving object M. More concretely, the detecting block calculates the horizontal image angle α_i, α_j and α_k of markers P_i, P_j and P_k of which image was extracted by the image processing unit 1101 by using equation 2. Then, referring to the position information of the markers P_i, P_j and P_k that were stored in the marker DB 1000 of the global coordinate system 303 (for example, latitude, longitude, or orthogonal coordinate value on setting the reference point), the detecting block extracts the marker data corresponding to the observed markers P_i, P_j and P_k from the marker DB 1000.

Since it is difficult to identify the markers P_i, P_j and P_k when the position and direction of the moving object M are unknown, the detecting unit 1102 identifies the position of image processed markers in the global coordinate system 303 by estimating the longitude/latitude and the marker type of the extracted markers P_i, P_j and P_k, by extracting the nearest marker data from the marker DB 1000, by using the output result of the GPS 913 as the position of the moving object M and by using the measured value of the electronic compass 914 as the direction of moving object M.

In the case that position measurement is continuously executed, it is also possible to use the previous position data of the vehicle itself measured by the moving object locating system 1100 at a previous time. In case of utilizing an assumed position and direction of vehicle itself, the detecting unit 1102 extracts the markers existing in horizontal direction angle α_i, α_j and α_k, of each camera CMi, CMj and CMk when the moving object M is arranged in its position and direction. And then, it extracts the markers P_i, P_j and P_k with the nearest distance and having the smallest angle error from the extracted markers P_i, P_j and P_k as the corresponding markers (coordinate values P_i*, P_j*, and P_k* in the global coordinate system 303).

Marker data D_i={α_i,β_i, (t_ix, t_iy),P_i*},D_j={α_j,β_j, (t_jx, t_jy),P_j*},D_k={(α_k,β_k, (t_kx, t_ky),P_k*} of the extracted marker P_i*, P_j* and P_k* are output to the identifying unit 1103.

The identifying unit 1103 identifies the moving object origin trajectory C_ij* relating to an unknown origin position of the moving object M according to the markers P_i* and P_j* (marker data D_i, marker data D_j) detected by the detecting unit 1102. It also identifies the moving object origin trajectory C_jk* relating to an unknown origin position of the moving object according to the markers P_j* and P_k* (marker data D_j, marker data D_k). A detailed functional structure of the identifying unit 1103 is described later (refer to FIG. 12).

The locating unit 1104 locates the position of an unknown moving object origin to a coordinate of an intersecting point of trajectories of moving object origin C_ij* and C_jk* identified by the identifying unit 1103 by using equations 10 and 11. It also locates the azimuthal angle ω(μ_ij) of the moving object M by using equations 12.1 and 12.2. That is, the locating unit 1104 outputs the coordinate of the intersecting point of trajectories of the moving object origin C_ij* and C_jk* and the azimuthal angle ω(μ_ij) of the moving object M as the output data 1110.

Next, a functional structure of the identifying unit 1103 is described in detail. FIG. 12 is a block diagram showing the identifying unit 1103. The identifying unit 1103 includes a virtual optical center calculation unit 1131 ( . . . , 1131ij, 1131j k, . . . ), a inter-marker angle calculation unit 1132 ( . . . , 1132ij, 1132j k, . . . ), a marker circle calculation unit 1133 ( . . . , 1133ij, 1133j k, . . . ) and a function decision unit 1134 ( . . . , 1134ij, 1134j k, . . . ).

Here, a case in which the identifying block reads two marker data D_i and D_j is explained. As for the case that the identifying block read two marker data D_j and D_k, a description is skipped because the same explanation can be applied by just replacing the suffix “ij” with “jk”.

First, the virtual optical center calculation unit 1131ij acquires two marker data D_i and D_j and calculates the virtual optical center Q_ij by using equation 4.1 to 4.2. Using equation 5, the inter-marker angle calculation unit 1132_ij calculates the inter-marker angle θ_ij* of the two markers P_i and P_j as viewed from the virtual optical center Q_ij that was calculated by the virtual optical center calculation unit 1131_ij.

Using equations 7.1 and 7.2, the marker circle calculation unit 1133ij calculates the center position O_ij* and radius r_ij* of a marker circle E_ij for identifying the position of the virtual optical center O_ij estimated from the inter-marker angle θ_ij* calculated by the inter-marker angle calculation unit 1132_ij in the global coordinate system 303.

Using equations 8 and 9, the function decision unit 1134_ij decides the origin trajectory function C_ij*(μ_ij) of the moving object M in the global coordinate system 303 (refer to equation 10. In this way, the function decision unit 1134 decides the origin trajectory function C_ij*(μ_ij) and C_jk* (μ_jk) for each marker pair (P_i, P_j) and (P_j, P_k)

In this case, the locating unit 1104 calculates (λ_ij, μ_jk) that makes the evaluation function F(μ_ij, μ_jk) “0” or minimizes it by using the origin trajectory function C_ij* (μ_ij) calculated from one marker pair (P_i, P_j) and the origin trajectory function C_jk* (μ_jk) calculated from another marker pair (P_j, P_k). Then, by substituting the calculated (μ_ij, μ_jk) into equations 9 and 10, position of moving object origin can be obtained, and by substituting the calculated (μ_ij, μ_jk) into equations 11 and 12, the azimuthal angle as a direction of the moving object M can be obtained.

In order to calculate (μ_ij, μ_jk) that minimizes the evaluation function F(μ_ij, μ_jk), it is also possible to use exploratory numerical analysis method such as a steepest descent method. In this case, because whether the optimum solution can be obtained depends on what initial value is selected, the number of exploratory repletion greatly changes depending on the initial value. Therefore, it is also possible to use the values derived from the position and the azimuthal angle obtained from the GPS 913 and the electronic compass 914 as the initial value of exploring. In case that position measurement is executed continuously, it is possible to start exploration using the value derived from the position and the azimuthal angle that have already been obtained during a previous measurement or the previous value of (μ_ij, μ_jk) itself.

FIG. 13 is a flowchart showing a procedure of moving object locating system according to an embodiment of this invention. As shown in FIG. 13, the GPS 913 obtains the initial position of the moving object M and the electronic compass 914 obtains the azimuthal angle of the moving object M (step S1301).

Then, the image processing unit 1101 extracts marker images from images photographed by the camera CM_i (step S1302). The detecting unit 1102 calculates the horizontal image angle α_i (step S1303) and detects the marker data D_i, D_j and D_k of each marker P_i*, P_j* and P_k* in the global coordination system 303 (step S1304).

Next, the identifying unit 1103 first calculates the marker line L_i of marker P_i* and the marker line L_j of the marker P_j* by using equation 3 (step S1305), and the virtual optical center calculation unit 1131 calculates the virtual optical center Q_ij that is an intersecting point of the marker line Li and L_j by using equations 4.1 to 4.2 (step S1306).

Then, the inter-marker angle calculation unit 1132ij calculates the inter-marker angle θ_ij* of two markers P_i and P_j as viewed from the virtual optical center Q_ij calculated by the virtual optical center calculation unit 1131 by using equation 5 (step S1307).

By using equations 7.1 and 7.2, the Marker circle calculation unit 1133ij calculates the center position O_ij* and the radius r_ij* of the marker circle E_ij to identify the position of the virtual optical center Q_ij in the global coordinate system 303 that is estimated from the inter-marker angle θ_ij* calculated by the inter-marker angle calculation unit 1132_ij (step S1307).

The function decision unit 1134_ij calculates the inter-marker angle φ_im between one marker P_i and the moving object origin O_m looked at from the virtual optical center Q_ij by using equations 6.1 to 6.3, and also calculates the inter-marker angle φ_jm between the marker P_j and the moving object origin O_m looked at from the virtual optical center Q_ij by using equations 6.2 to 6.5 (step S1309).

Where, the processes from step S1305 to step S1309 are executed similarly for the marker data D_j and D_k of the marker P_j* and P_k* using the inter-marker angle calculation unit 1132_jk, the marker circle calculation unit 1133jk and the function decision unit 1134jk.

After this, the function decision unit 1134 executes the variable decision process (step S1310). That is, it decides the origin trajectory function C_ij*(μ_ij), C_jk*(μ_jk), . . . for each marker pair (P_i, P_j), (P_j, P_k), . . . by using equations 8 and 9. Then, by substituting the origin trajectory function C_ij*(μ_ij), C_jk*(μ_jk) into equation 11, the function decision unit 1134 decides the variables (μ_ij, μ_jk) that makes evaluation function F(μ_ij, μ_jk) “0” or minimizes it.

The locating unit 1104 calculates the moving object origin position C_ij* in the global coordinate system 303 by substituting (μ_ij, μ_jk) into equations 9 and 10 (step S1311) And the locating unit 1104 calculates also the azimuthal angle ω(μ_ij) as the direction of the moving object M by substituting (μ_ij, μ_jk) into equations 12.1 and 12.2 (step S1312).

After this, whether to continue moving object-locating is judged (step S1313). When moving object-locating is continued (step S1313: Yes), the process returns to step S1302. In this case, calculation of a new position and azimuthal angle of the moving object origin is executed by using the moving object origin position C_ij* and the azimuthal angle ω(μ_ij) calculated in Step S1311 and Step S1312. On the other hand, in case that moving object-locating is not continued (step S1313: No), the procedure of moving object-locating is completed.

FIG. 14 is a flowchart showing a concrete procedure of variable decision process (step S1310).

First, it is assumed that an initial value of the variable (μ_ij, μ_jk) corresponds to a focused point (λ_ij, λ_jk) and its step width is Δλ (step S1401). Then, the focused point (λ_ij,λ_jk) is substituted into the evaluation function F and, at the same time, values neighboring the focused point (λ_ij,λ_jk) (in this case, neighboring four points) are substituted into the evaluation function F (step S1402).

FIG. 15 is a diagram showing the neighboring four points to the focused point (λ_ij,λ_jk). As shown in FIG. 15, the neighboring four points (indicated by reference characters 1501 to 1504) to the focused point (λ_ij,λ_jk) indicated by 1500 becomes a value increased/decreased by the step width Δλ according to the coordinate system of evaluation function F shown in FIG. 8.

As shown in FIG. 14, whether all of the values of the evaluation function F corresponding to the neighboring four points are greater than a value of the evaluation function F of the focused point (λ_ij,λ_jk) is judged (step S1403). In case that any one out of four values of the evaluation function F corresponding to the neighboring four points is equal to or less than the value of the evaluation function F (λ_ij,λ_jk) (step S1403: No), the process sets the coordinate value having the minimum value among values of evaluation function F corresponding to the four neighboring points (any one of 1501 to 1504) to a new focused point (λ_ij,λ_jk) (step S1404), and returns to step S1402.

On the other hand, in case that all of the values of the evaluation function F corresponding to the neighboring four points is greater than the value of the evaluation function F of the focused point (λ_ij,λ_jk) (step S1403: Yes), whether the step width Δλ is less than a threshold value λt (for example, λt=π/1000) is judged (step S1405).

When the step width Δλ is equal to or greater than the threshold value λt (step S1405: No), the process makes the step width Δλ small to ½ (step S1406) and returns to step S1402. On the other hand, if the step widthΔλ was smaller than the threshold value λt (step S1405: Yes), the process decides to take the finally obtained focused point (λ_ij,λ_jk) as the value of the variable (μ_ij, μ_jk) (step S1407)

In this way, the process is completed judging that a sufficiently accurate solution was obtained. Although this variable decision process is a numerical analysis method in the angle space of μ, even if this is done in the same orthogonal coordinate space as the global coordinate system 303, if minimization of the evaluation function F is executed as the result, the same result is obtained.

A moving object locating method described in a form of implementation of this invention can be realized by executing a prepared program on a computer such as a personal computer or a work station. This program is recorded on readable storage media such as a hard disk, a flexible disk, a CD-ROM, an MO, a DVD that can be read by a computer, and is executed by being loaded to a computer from these recording media. This program can be a transmission media that can be delivered to users via networks such as the internet.

According to the embodiments described above, it is possible to increase the camera installation flexibility on a moving object as well as to increase accuracy in locating the moving object.

Although the invention has been described with respect to a specific embodiment for a complete and clear disclosure, the appended claims are not to be thus limited but are to be construed as embodying all modifications and alternative constructions that may occur to one skilled in the art which fairly fall within the basic teaching herein set forth.

Claims

1. A computer-readable recording medium that stores therein a program for locating a moving object by using a plurality of images taken outside the moving object with a plurality of cameras mounted on the moving object, the program causing a computer execute: detecting a position and a direction for a first marker, a second marker, and a third marker contained in the images;identifying a first origin trajectory relating to an unknown origin position for the moving object based on the first marker and the second marker;identifying a second origin trajectory relating to the unknown origin position based on the second marker and the third marker; andlocating the unknown origin position at coordinates of an intersecting point of the first origin trajectory and the second origin trajectory.

2. The computer-readable recording medium according to claim 1, wherein the program further causes the computer to execute: calculating a first virtual optical center having a field of vision that includes the first marker and the second marker, and a second virtual optical center having a field of vision that includes the second marker and the third marker;calculating a first inter-marker angle between the first marker and the second marker having a vertex at the first virtual optical center and a second inter-marker angle between the second marker and the third marker having a vertex at the second virtual optical center; andidentifying the first origin trajectory such that the first inter-marker angle, a distance between the first virtual optical center and the unknown origin position, and an angle formed between a line from the first virtual optical center to the second marker and a line from the first virtual optical center to the unknown origin position remain constant;identifying the second origin trajectory such that the second inter-marker angle, a distance between the second virtual optical center and the unknown origin position, and an angle formed between a line from the second virtual optical center to the third marker and a line from the first virtual optical center to the unknown origin position remain constant.

3. The computer-readable recording medium according to claim 2, wherein the program further causes the computer to execute: deciding a first variable and the a second variable such that an evaluation function that evaluates a difference by numerical analysis is minimized, the difference between a function expressing the first origin trajectory and a function expressing the second origin trajectory when a first function that expresses the first origin trajectory is defined by using the first variable and a second function that expresses the second origin trajectory is defined by using the second variable;locating the unknown origin position at the intersecting point by assigning the first variable to the first function and by assigning the second variable to the second function.

4. The computer-readable recording medium according to claim 3, wherein when a first marker circle function is defined by using the first variable that expresses the first marker circle on which the first marker, the second marker, and the first virtual optical center exist, and a second marker circle function is defined by using the second variable that expresses a second marker circle on which the second marker, the third marker, and the second virtual optical center exist, the locating includes locating an azimuthal angle of the moving object by assigning the first variable to the first marker circle function and by assigning the second variable to the second marker circle function.

5. The computer-readable recording medium according to claim 4, wherein the program further causes the computer to execute repeating the detecting, the identifying, and the locating using the moving object origin position and the azimuth direction, when the position and the direction of the first marker, the second marker, and the third marker are detected by using the position and the azimuth direction of the moving object obtained from a global positioning system and an electronic compass mounted on the moving object.

6. An apparatus for locating a moving object by using a plurality of images taken outside the moving object with a plurality of cameras mounted on the moving object, the apparatus comprising: a detecting unit that detects a position and a direction for a first marker, a second marker, and a third marker contained in the images;a first identifying unit that identifies a first origin trajectory relating to an unknown origin position for the moving object based on the first marker and the second marker;a second identifying unit that identifies a second origin trajectory relating to the unknown origin position based on the second marker and the third marker; anda locating unit that locates the unknown origin position at coordinates of an intersecting point of the first origin trajectory and the second origin trajectory.

7. A method of locating a moving object by using a plurality of images taken outside the moving object with a plurality of cameras mounted on the moving object, the method comprising: detecting a position and a direction for a first marker, a second marker, and a third marker contained in the images;identifying a first origin trajectory relating to an unknown origin position for the moving object based on the first marker and the second marker;identifying a second origin trajectory relating to the unknown origin position based on the second marker and the third marker; andlocating the unknown origin position at coordinates of an intersecting point of the first origin trajectory and the second origin trajectory.