Patent Application
20060204038
The present invention relates to a stereo camera apparatus and a stereo image process technique, and particularly relates to a stereo camera apparatus in which imaging timing and image input timing are adjusted in a stereo image process.
A stereo camera apparatus in which a distance to an object is calculated using a pair of images taken by two imaging units, and the object is recognized based on the calculated distance has begun to be applied to a supervisory system for detecting entry of a suspicious person or abnormality or a vehicle mounted system for assisting a driver to drive safely.
The stereo image process used in the supervisory system or the vehicle mounted system is to apply a triangular surveying technique to a pair of images taken at a positional interval so as to obtain a distance. The stereo image processing apparatus typically has a pair of imaging units and a stereo image processing LSI for applying a triangular surveying process to a pair of images output by the imaging units. The stereo image processing LSI performs a process to superimpose pixel information included in the pair of images so as to obtain a position where the two images coincide with each other. Thus, the stereo image processing LSI achieves the triangular surveying process. It will be ideal if there is no misalignment but a parallax between the pair of images. It is therefore necessary to adjust each imaging unit to eliminate any variation in optical properties or signal properties.
Particularly in a vehicle mounted environment, there is an application demand to detect, for example, a vehicle, a human being, an obstacle, etc. in front and to deal therewith safely in advance. It is therefore necessary to make sure to measure a distance to a far object and recognize the object.
Generally in the stereo camera apparatus, the aforementioned request to have no misalignment but a parallax becomes stronger with increase of the distance to the object.
Z=bf/δ (1)
By use of the expression (1) and on the assumption that it is, for example, aimed at use in a vehicle mounted environment and a distance to be measured is 100 m (Z=100 m: to measure a distance to an object which is 100 m ahead), there occurs a lag between the timings at which data of taken images are propagated to the image processing LSI respectively. As a result, there occurs a horizontal one-pixel misalignment between the two images as shown in
First, based on the conditions to arrange a typical stereo camera apparatus, assume that parameter values to be assigned to the expression (1) except the value Z are set as f=10 mm and b=350 mm, and imaging devices to be used are ¼-inch 380-thousand-pixel CCD (Charge Coupled Device) image sensors. Assign these parameter values to the expression (1). Then, deterioration of about 20 m can occur in the distance measuring accuracy if there occurs a horizontal one-pixel misalignment as shown in
This distance measuring result is out of an allowable range of the required accuracy in measuring the distance of 100 m. It is therefore necessary to make a design not to produce a lag (horizontal pixel misalignment) between possible timings to propagate data of taken images to the image processing LSI respectively when the stereo camera apparatus is used with the aforementioned parameter values. Further, when the stereo image processing system is used in a vehicle mounted system, it is also requested to make the stereo image processing system smaller in size, lower in price and higher in reliability.
Also as for the timing to take each image, it is necessary to be careful not to produce any misalignment in a vehicle mounted environment because the image is taken when the vehicle is moving.
Measures against the aforementioned misalignment have been taken in the background art as follows. First, misalignment (horizontal pixel misalignment) as to possible timing at which data of each taken image is propagated to the image processing LSI is generally dealt with by a so-called equal-length wiring technique. In the equal-length wiring technique, the lengths of signal lines between respective imaging units 301 and 302 and an image processing LSI 303 for processing images output by the imaging units 301 and 302 are made equal to each other, as shown in a configuration example of
In the configuration example of
In this event, the wire length of a signal (imaging control signal 306) between the imaging unit A 301 and the timing generator 307 is made equal to the wire length of the signal (imaging control signal 306) between the imaging unit B 302 and the timing generator 307. Thus, misalignment is prevented from occurring in the imaging timing at which each imaging unit takes an image and the timing at which each imaging unit transfers the taken image. In addition, the wire length of a signal (image data signal 304) between the imaging unit A 301 and the image processing LSI 303 is made equal to the wire length of a signal (image data signal 305) between the imaging unit B 302 and the image processing LSI 303. Thus, horizontal pixel misalignment is prevented from occurring.
However, in the method shown in
In addition thereto, it is expected that the image size will increase with improvement of throughput. When the sampling rate of image data to be transferred is increased thus, noise or temperature will have a greater influence in the method based on equal-length wiring. When such a situation is taken into consideration, probable occurrence of pixel misalignment as shown in
Further, in the method based on equal-length wiring, the way of setting a board configuration is restricted. Whenever a new board is designed or whenever an imaging unit to be used is changed, the aforementioned influence of noise or temperature has to be reviewed. Thus, there is also a problem in attaining a system smaller in size and lower in price.
A solution to the aforementioned problems in equal-length wiring has been proposed, for example, as disclosed in JP-A-6-70216 or JP-A-8-331548. JP-A-6-70216 proposes a method in which it is not necessary to adjust any circuit even if the cable length between each imaging unit and an image processing unit is changed when a new board is developed or the imaging unit is changed. According to this method, a mechanism for detecting the cable length and a mechanism for correcting a propagation delay time error caused by the cable length are provided in the image processing unit to be able to perform signal processing while correcting the propagation delay time error.
On the other hand, JP-A-8-331548 proposes a method in which a phase difference between a sync signal and a video signal is eliminated even when the cable length is long. According to this method, in a television camera control system constituted by a television camera and a control apparatus, a delay may occur between a sync signal and a video signal due to a long cable length between the television camera and the control apparatus. In such a case, a command to set the phase of the sync signal forward by a predetermined amount in accordance with the cable length is sent from the control apparatus to the television camera. Thus, the phase difference caused by the delay is canceled. In this event, the proposed television camera control system is designed as follows. That is, the television camera performs an imaging process and outputs a video signal to the control apparatus by use of a sync signal generated by the control apparatus. The control apparatus processes the input video signal in sync with the sync signal.
Next, a solution to misalignment as to the timing at which each image is taken has been proposed, for example, as disclosed in JP-A-2002-165108. A configuration example of JP-A-2002-165108 is shown in
Of the background-art techniques concerning misregistration as to the timing at which data of taken images are propagated to an image processing LSI, the technique disclosed in JP-A-6-70216 can absorb the propagation delay characteristic caused by the cable length or cables themselves when a new board is designed. However, a correction process is executed after phase comparison. It is therefore impossible to absorb the influence caused by noise or a temperature difference in real time after each unit has been mounted on the board. In addition, it cannot be denied that it is likely that pixel misalignment as shown in
Also in JP-A-8-331548, it is possible to eliminate the phase difference between a sync signal used in one imaging unit and a video signal output by the imaging unit. However, when a plurality of such imaging units are provided, it cannot be guaranteed that no pixel alignment as shown in
On the other hand, according to JP-A-2002-165108 proposing a solution to misalignment as to timing to take images, the external trigger signal generating unit 407 and the clock signal generating unit 408 are disposed so closely to the respective imaging units 401-403 that the imaging timings of the imaging units 401-403 can be synchronized. In the same manner, the memories 409-411 are disposed so closely to the imaging units respectively that it is also possible to deal with misalignment (horizontal pixel misalignment) as to timing to propagate data of taken images to the image processing LSI. However, even according to this method, it is difficult to design a board so as to dispose the external trigger signal generating unit 407 or the clock signal generating unit 408, for example, closely to both the first imaging unit 401 and the second imaging unit 402 when there is a long distance between the two imaging units.
As a result, there also arises an occasion when the wire length between the first imaging unit 401 and the external trigger signal generating unit 407 and the wire length between the first imaging unit 401 and the clock signal generating unit 408 must be careful to be equal to the wire length between the second imaging unit 402 and the external trigger signal generating unit 407 and the wire length between the second imaging unit 402 and the clock signal generating unit 408 respectively. It is therefore impossible to solve the aforementioned problem as to the imaging timing with accuracy required in a vehicle mounted environment. In addition, due to the restriction placed on dispositions in a board, the degree of freedom in design is spoiled. Further, also as to horizontal pixel misalignment, memories have to be placed for the imaging units individually. It is therefore difficult to attain the requirements while making the system smaller in size and lower in price.
An object of the present invention is to provide a vehicle mounted stereo camera apparatus which can solve the foregoing problems belonging to the background art.
Another object of the present invention is to provide a vehicle mounted stereo camera apparatus in which a stereo image process can be performed by an image processing LSI using image data output by one imaging unit and image data output by another imaging unit so as to prevent occurrence of pixel misalignment between the image data.
A further object of the present invention is to provide a vehicle mounted stereo camera apparatus which can be made small in size as a whole, low in price and high in reliability without any misalignment even among the imaging timings of a plurality of imaging units.
In order to attain the foregoing objects, the present invention is chiefly configured as follows.
A vehicle mounted stereo camera apparatus including a plurality of imaging units for generating image data sequentially based on optical information from an object, and an image processing unit for importing the image data generated by the plurality of imaging units respectively, processing the imported image data individually, and then performing a stereo image process operation is designed as:
the image data input into the image processing unit are synchronized with clock signals for importing and processing the image data respectively, so as to perform the stereo image process operation while eliminating horizontal pixel misalignment from the image data from the plurality of imaging units.
In addition, in the vehicle mounted stereo camera apparatus, analog-to-digital conversion units for converting analog data into digital data are provided for the imaging units and between the imaging units and the image processing unit respectively; and
clock signals for analog-to-digital conversion used in the analog-to-digital conversion units are used as clock signals for importing and processing image data from the imaging units in the image processing unit.
Further, a vehicle mounted stereo camera apparatus including a plurality of imaging units for generating image data sequentially based on optical information from an object, and an image processing unit for importing the image data generated by the plurality of imaging units respectively, processing the imported image data individually, and then performing a stereo image process operation is designed as:
the image processing unit includes an imaging timing misalignment detection unit for detecting misalignment as to imaging timing among the imaging units; and
the misalignment as to imaging timing is adjusted by the misalignment detection unit and timing signals with which the imaging units are to take images are output to the imaging units respectively through a timing generator.
According to the present invention, a lag of timing (horizontal pixel misalignment) which may be generated when data of a plurality of taken images are propagated to the image processing LSI can be eliminated so that a stereo camera apparatus high in reliability and high in accuracy can be obtained.
In addition, it is not necessary to take it into consideration to set equal-length wiring as to lines between the respective imaging units and the image processing LSI. Accordingly, the system can be constructed with a minimum configuration, and the apparatus can be made small in size and low in price.
Other objects, features and advantages of the invention will become apparent from the following description of the embodiments of the invention taken in conjunction with the accompanying drawings.
Vehicle mounted stereo camera apparatus according to embodiments of the present invention will be described below in detail with reference to
A vehicle mounted stereo camera apparatus according to a first embodiment of the present invention will be described below with reference to
In
The imaging unit A 601 and the imaging unit B 602 are typically constituted by CCD (Charge Coupled Device) sensors. Each of the imaging unit A 601 and the imaging unit B 602 converts external light, that is, optical information from an object into electric charge information, and supplies one screen of image data sequentially to the image processing LSI 603 through the ADC 606, 607 for generating digital image data. In this event, the imaging unit A 601 and the imaging unit B 602 receive an imaging timing signal 604 for imaging, from the timing generator (TG) 605, and perform an imaging process in sync with the imaging timing signal 604.
Each ADC 606, 607 receives a clock reference signal for AD conversion, from the timing generator TG 605, and performs analog-to-digital conversion upon the imaged data. In that event, the ADC 606, 607 supplies the image processing LSI 603 with an analog-digital clock signal (hereinafter referred to as “AD clock signal”) 608, 609 used for converting the imaged data into digital data. The AD clock signal 608, 609 is generally named ADCLK, ADCK etc.
The image processing LSI 603 receives the AD clock signal 608, 609 from the ADC 606, 607 connected to the imaging unit 601, 602. The image processing LSI 603 adjusts the phase of the clock by means of a PLL 610, 611 provided in the image processing LSI 603, and samples the imaged data output from the ADC 606, 607.
Here, in the ADC 606, 607 which is a processing stage just before the image processing LSI 603, each AD clock signal 608, 609 is output in sync with image data A, B output by each imaging unit 601, 602. Accordingly, the image processing LSI 603 can import the image data into the memory 614 surely without any pixel misalignment.
More specifically, the image data A output from the ADC 606 are subjected to an A/D conversion process based on the AD clock signal 608. Therefore, the image data A are synchronized with the AD clock signal 608. On the other hand, the image data B output from the ADC 607 is synchronized with the AD clock signal 609. Accordingly, data import, image preprocessing and direct memory access controller (DMAC) operation of the image data A and the image data B are executed in accordance with the AD clock signal 608 and the AD clock signal 609 respectively. Thus, the image data A and the image data B are once stored in the memory 614 in the state where there is no pixel misalignment between the image data A and the image data B. In such a manner, no pixel misalignment as shown in
The image processing LSI 603 processes each image data individually and in sync with a PLL output clock signal based on the AD clock signal 608, 609, and accumulates the processed image data in the memory 614 through a DMAC 612, 613. Image preprocessing in the stage followed by the DMAC 612, 613 shown in
After that, a pair of image data accumulated in the memory 614 are read through a DMAC 615, and a stereo image process is performed thereon. In this event, a fast clock signal 617 is used for the DMAC 615 and a stereo image process 616 so that the processing time taken for the stereo image process can be shortened.
The image processing LSI 603 processes the image data A and B asynchronous and individually using the AD clocks used for A/D conversion respectively. Here, the details of the processing include an image import process, a gamma correction process, etc. Next, the image data processed independently of each other are transferred to the memory 614 through their corresponding DMAC channels respectively. Further, a pair of image data to be subjected to stereo processing are read from the memory 614 to the DMAC 615 using the fast clock 617. A stereo image process is performed using the same fast clock 617 as the DMAC 615. Thus, the flow of a series of operations is terminated.
In the configuration example shown in
Here, according to another configuration example of the first embodiment, which is not shown, two clock generators asynchronous with each other and independent of each other may be provided so that one of the clock generators supplies timing signals to the imaging units 601 and 602 while the other supplies timing signals to the ADCs 606 and 607. Also in this configuration example, in the same manner as in the aforementioned operation in the first embodiment, there is no fear that there occurs a pixel misalignment between image data A and B. According to a further configuration example of the first embodiment, three clock generators asynchronous with one another and independent of one another may be provided so that the first one of the clock generators supplies timing signals to the imaging units 601 and 602, and the second one supplies a timing signal to the ADC 606 while the third one supplies a timing signal to the ADC 607. Also in this configuration example, image data and an AD clock signal output from each ADC are synchronized with each other. Thus, there is no fear that there occurs a pixel misalignment between image data A and B.
As described above, according to this embodiment, the timing misalignment (horizontal pixel misalignment) which might occur when a plurality of pieces of data of taken images are propagated to the image processing LSI can be eliminated so that a high-reliability and high-accuracy stereo camera apparatus can be obtained. In addition, it is not necessary to make the cable lengths between the respective imaging units and the image processing LSI equal to each other. Accordingly, the system can be constructed with a minimum configuration, and the apparatus can be made small in size and low in price.
A vehicle mounted stereo camera apparatus according to a second embodiment of the present invention will be described below with reference to FIGS. 3 to 7.
In
In the imaging timing misalignment detection mechanism 813 shown in
In this event, the misalignment is detected in the following method. That is, XOR 901 is obtained from the ½ divided clock signal 810 output from the imaging unit A 801 and the ½ divided clock signal 811 output from the imaging unit B 802. The XOR 901 is counted by a counter unit 902. When the counted number exceeds a predetermined counter threshold value for detecting misalignment, an imaging timing signal inversion instruction unit 903 turns on an instruction signal 906 to invert the imaging timing signal 805 to be supplied to the imaging unit B 802. When the instruction signal 906 is turned on, the timing generator (TG) 812 inverts the reference clock signal to be supplied to the imaging unit B 802, and the imaging timing signal 805 incidental thereto.
As shown in the timing charts of
The resolution to detect the misalignment in imaging timing is doubled for the following reason. That is, in the misalignment detection mechanism shown in
Next, in an application example of the vehicle mounted stereo camera apparatus according to the second embodiment, a vehicle mounted stereo camera apparatus can be formed by applying an imaging timing misalignment detection mechanism as shown in
A vehicle mounted stereo camera apparatus according to a third embodiment of the present invention will be described below with reference to
In vehicle mounted stereo camera apparatus according to the third embodiment, not CCD (Charge Coupled Device) sensors but CMOS (Complementary Metal-Oxide Semiconductor) sensors 130 are used in imaging units. As shown in the configuration example of
On the other hand, in the configuration example of
In a system mode according to the third embodiment, images from the imaging units can be processed individually and independently. Accordingly, as shown in
A vehicle mounted stereo camera apparatus according to a fourth embodiment of the present invention will be described below with reference to
In
On the other hand, in
As described above, according to the present invention, misalignment as to imaging timing among respective imaging units and misalignment as to timing for the respective imaging units to output image data to an image processing LSI can be adjusted. Thus, a stereo matching process high in accuracy can be attained in a high-reliability method. In addition, according to the present invention, a vehicle mounted stereo camera apparatus can be obtained without additional parts and in a form high in degree of freedom in board design. Accordingly, the present invention can contribute to making the vehicle mounted stereo camera apparatus lower in price and smaller in size.
It should be further understood by those skilled in the art that although the foregoing description has been made on embodiments of the invention, the invention is not limited thereto and various changes and modifications may be made without departing from the spirit of the invention and the scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2005-011791 | Jan 2005 | JP | national |
