The present invention is related to a system and method of stereo mapping and in particular a system and method of stereo mapping using two cameras having different resolutions.
Stereo mapping is a system involving two cameras that allows camera images to be processed to determine information that may be difficult to obtain using a single camera. For instance, stereo mapping allows depth of an object to be determined utilizing the offset geometry of the camera images. In particular, epipolar geometrical matrices may be used to determine the depth of an object which can be used by an active control system of an automotive vehicle process a collision warning or make a corrective action.
Currently, stereo mapping systems require two cameras of the same resolution. Thus, the cameras are configured to provide similar fields of view, which limit the mapping capabilities. Thus, additional cameras are required to capture a greater field of view which adds costs to vehicle production. Accordingly, it remains desirable to have a stereo mapping system wherein multiple cameras of different fields of view may be used so as to reduce production costs.
A stereo mapping system and method for use in an automotive vehicle is provided. The stereo mapping system and method is configured to provide three dimensional distance of an object relative to the automotive vehicle. The system includes a first camera and a second camera. The first camera is configured to detect objects within an intermediate and long range distance of the automotive vehicle. The second camera is configured to detect objects within a short range of the automotive vehicle, relative to the first camera. The first camera has a first camera resolution and a first field of view. The second camera has a second camera resolution and a second field of view. The second camera resolution is different than the first camera resolution. The second field of view is wider than the first field of view.
The system further includes an image processor. The image processor includes a first processing segment and a second processing segment. The first processing segment is configured to process images of the first camera so as to detect an object. In particular, the first processing segment captures an image frame at the end of a processing period and processes the used image frame so as to detect an image for a plurality of used frames, whereas the remaining frames of the first camera are unused frames.
The second processing segment is configured to process the second camera. The second processing segment processes a corresponding frame of the second camera. The corresponding frame corresponds in time to an unused frame of the first camera. The second processing segment processes the corresponding frame so as to match the resolution of corresponding frame with the unused frame of the first camera.
The system further includes an image mapping segment. The image mapping segment matches a predetermined pixel area of the unused frame of the first camera with a corresponding pixel area of the corresponding frame of the second camera. The image processor is further configured to process the unused frame of the first camera with the corresponding frame of the second camera so as to determine depth and distance of objects within the predetermined pixel area of the unused frame and the corresponding frame.
A method configured to process camera film from a first camera and a second camera so as to perform stereo mapping in an automotive vehicle is also provided. The method includes the steps of providing a first camera having a first camera resolution. The first camera is configured to detect objects within an intermediate and long range. The second camera is configured to detect objects within a short range, relative to the first camera. The first camera has a first camera resolution and a first field of view. The second camera has a second camera resolution and a second field of view, wherein the second camera resolution is different than the first camera resolution. The second field of view is wider than the first field of view.
The method includes the step of providing an image processor. The image processor includes a first processing segment and a second processing segment. The first processing segment is configured to process images of the first camera so as to detect an object. In particular, the first processing segment captures an image frame at the end of a processing period and processes the used image frame so as to detect an image for a plurality of used frames, whereas the remaining frames of the first camera are unused frames.
The second processing segment is configured to process the second camera. The second processing segment processes a corresponding frame of the second camera, the corresponding frame corresponding in time to an unused frame of the first camera, and wherein the second processing segment processes the corresponding frame so as to match the resolution of corresponding frame with the unused frame of the first camera.
The method further includes the step of providing an image mapping segment. The image mapping segment matches a predetermined pixel area of the unused frame of the first camera with a corresponding pixel area of the corresponding frame of the second camera, wherein the image processor is further configured to process the unused frame of the first camera with the corresponding frame of the second camera so as to determine depth and distance of objects within the predetermined pixel area of the unused frame and the corresponding frame.
The embodiments set forth in the drawings are illustrative and exemplary in nature and not intended to limit the subject matter defined by the claims. The following detailed description of the illustrative embodiments can be better understood when read in conjunction with the following drawings where like structure is indicated with like reference numerals and in which:
A stereo mapping system 10 and method 100 for use in an automotive vehicle 200 is provided. In particular, the stereo mapping system 10 and method 100 is configured to perform stereo mapping of cameras having different resolutions and different fields of view. The stereo mapping system 10 and method 100 is configured to provide three dimensional distance of an object relative to the automotive vehicle 200.
With reference now to
The first camera 12 is mounted on the automotive vehicle 200 so as to be disposed on a vertical and horizontal plane different than that of the second camera 14. For example, the first camera 12 may be mounted so as to be elevated above the second camera 14, and behind the second camera 14. Thus, the images taken from each camera 12, 14 are provided at different camera angles which allows the images to have offset geometries with respect to each other. The offset geometries of the images may then be used to determine information such as distance and depth utilizing the concepts of epipolar geometry.
The first camera 12 is illustratively shown fixedly mounted to the upper portion of the windshield 210 relative to the second camera 14 and is configured to detect objects within an intermediate and long range. The first camera 12 has a first camera resolution 16 and a first field of view 18.
The second camera 14 has a second camera resolution 20 and a second field of view 22. The second camera resolution 20 is different than the first camera resolution 16. The second camera 14 is illustratively shown mounted on the front bumper 220 of the automotive vehicle 200 and disposed beneath the first camera 12. The second camera 14 is configured for use with near object recognition, near being relative to the first camera 12 which is configured for use with object recognition for objects greater in distance than the second camera 12.
The second field of view 22 is wider than the first field of view 18 and spatial resolution of the second camera 14 is greater than the spatial resolution of the first camera 12. For illustrative purposes, the first camera 12 has a resolution of 1080p whereas the second camera 14 has a resolution of 752p, and the first camera 12 is configured to provide images for object detection of objects out to 150 m with a camera angle of 50 degrees, whereas the second camera 14 is configured is configured to provide images for object detection of objects out to 50 m with a camera angle of 180 degrees.
With reference again to
With reference again to
The second processing segment 28 is configured to process images from the second camera 14. The second processing segment 28 processes a corresponding frame 34 of the second camera 14. For use herein, a corresponding frame 34 refers to an image frame taken by the second camera 14 that corresponds in time to an unused frame 32 of the first camera 12. The second processing segment 28 is further configured to process the corresponding frame 34 so as to match the resolution of corresponding frame 34 with the unused frame 32 of the first camera 12.
With reference again to
The region of interest 38 may be determined by the coincidence, identified by the lined section, between the field of views of the first and second cameras 12, 14. The region of interest 38 of the corresponding frame 34 is the same pixel area as the region of interest 38 of the unused frame 32. Accordingly, the image mapping segment 36 may utilize epipolar rectification to determine image information, the concepts of epipolar rectification are provided in
The image processor 24 may be further configured to remove distortion from the cropped image. In particular, the image processor 24 may execute a software program that adjusts pixel information to remove distortion. It should be further appreciated that the image processor 24 may identify a region of interest 38, crop and remove distortion from the corresponding and unused frames 32 either before or after the second processing segment 28 matches the resolution of the corresponding frame 34 with the correlating unused frame 32. Preferably, the image processor 24 performs region of interest 38 identification, cropping and distortion removal prior to stereo mapping so as to reduce processing time.
The image processor 24 is further configured to process the unused frame 32 of the first camera 12 with the corresponding frame 34 of the second camera 14 so as to determine depth and distance of objects within the predetermined pixel area of the unused frame 32 and the corresponding frame 34. As discussed above, information such as depth, distance, and object recognition may be done by utilizing the concepts of epipolar geometry. Accordingly, information regarding range estimation of objects, road surface information and curb information may be obtained using cameras with two different resolutions.
Further, the image processor 24 is configured to alternate between a stereo mapping state and an object detection state. In the stereo mapping state the image mapping segment 36 matches the predetermined pixel area of the frame of the first camera 12 with a corresponding pixel area of the frame of the second camera 14 which correlates in time with the frame of the first camera 12. In the object detection state the image processor 24 processes the first camera 12 image to detect an object. Thus, in the instance of the object detection state, object detection is conducted and the information may be transmitted to an active control system 240 of the automotive vehicle 200. Whereas, in the instance of stereo mapping state, information about the image, such as depth, distance and the like may be determined. It should be appreciated that such information may be transmitted to the active control system to be processed with the detected object to execute an automotive vehicle 200 function such as steering, braking or the like.
With reference now to
With reference now to
The first camera 12 has a first camera resolution 16 and a first field of view 18. The second camera 14 has a second camera resolution 20 and a second field of view 22, wherein the second camera resolution 20 is different than the first camera resolution 16. The second field of view 22 is wider than the first field of view 18.
The second camera 14 has a second camera resolution 20 and a second field of view 22. The second camera resolution 20 is different than the first camera resolution 16. The second camera 14 is illustratively shown mounted on the front bumper 220 of the vehicle and disposed beneath the first camera 12. The second field of view 22 is wider than the first field of view 18 and has a shorter range relative to the first camera 12.
For illustrative purposes, the first camera 12 has a resolution of 1080p whereas the second camera 14 has a resolution of 752p, and the first camera 12 is configured to provide images for object detection of objects out to 150 m with a camera angle of 50 degrees, whereas the second camera 14 is configured is configured to provide images for object detection of objects out to 50 m with a camera angle of 180 degrees.
The first camera 12 and the second camera 14 may be mounted on the automotive vehicle 200 so as to be disposed on a vertical and horizontal plane different than that of each other, wherein the cameras are oriented along the same azimuth so as to gather the same image but taken from different angles. For example, the first camera 12 may be mounted so as to be elevated above the second camera 14, and behind the second camera 14. Thus, the images taken from each camera 12, 14 are provided at different camera angles which allows the images to have offset geometries with respect to each other. The offset geometries of the images may then be used to determine information such as distance and depth utilizing the concepts of epipolar rectification.
The method 100 includes the step 130, capturing an image frame from the first camera 12 so as to generate a used frame 30, wherein the remaining frames of the first camera 12 are unused frames 32. Step 130 may be executed by an image processor 24. The image processor 24 includes a first processing segment 26 and a second processing segment 28. The method 100 includes the step capturing an image frame from the first camera 12 and processing the used image frame so as to detect an object.
Step 130 may be executed by having the first processing segment 26 process images of the first camera 12 so as to detect an object. The first camera 12 image is transmitted to the image processor 24 wherein the image processor 24 executes the first processing segment 26 to capture image frames taken by the first camera 12 so as to generate a used image frame. The method 100 may further include step 160, transmitting the used image frame to an active control system 240 so as to perform a vehicle function, or generate a collision warning, or both.
The method 100 includes step 140, matching the resolution of a corresponding frame 34 of the second camera 14 with an unused frame 32 of the first camera 12, wherein the corresponding frame 34 corresponds in time with the unused frame 32. The step may be executed by the second processing segment 28. The second processing segment 28 processes a corresponding frame 34 of the second camera 14. For use herein, a corresponding frame 34 refers to an image frame taken by the second camera 14 that corresponds in time to an unused frame 32 of the first camera 12. The second processing segment 28 is further configured to process the corresponding frame 34 so as to match the resolution of corresponding frame 34 with the unused frame 32 of the first camera 12.
The method 100 further includes step 150, stereo matching of the corresponding frame 34 with the adjusted resolution and the unused frame 32 so as to obtain image information. The stereo matching step may be performed by an image mapping segment 36. The image mapping segment 36 matches a predetermined pixel area of the unused frame 32 of the first camera 12 with a corresponding pixel area of the corresponding frame 34 of the second camera 14. Accordingly, the image mapping segment 36 obtains an image of the same area of camera coverage, but taken from different views and having the same resolution.
The image processor 24 is further configured to process the unused frame 32 of the first camera 12 with the corresponding frame 34 of the second camera 14 so as to determine depth and distance of objects within the predetermined pixel area of the unused frame 32 and the corresponding frame 34. As discussed above, the information such as depth, distance, and object recognition may be done by utilizing the concepts of epipolar geometry. Accordingly, information regarding range estimation of objects, road surface information and curb information may be obtained using cameras with two different resolutions.
According to another aspect of the method 100, the image processor 24 may be configured to alternate between a stereo mapping state and an object detection state. In the stereo mapping state the image mapping segment 36 matches the predetermined pixel area of the a frame of the first camera 12 with a corresponding pixel area of the a frame of the second camera 14 which correlates in time with the frame of the first camera 12. In the object detection state the image processor 24 processes the first camera 12 image to detect an object. Thus, in the instance of the object detection state, object detection is conducted and the information may be transmitted to an active control system. Whereas, in the instance of stereo mapping state, information about the image, such as depth, distance and the like may be determined. It should be appreciated that such information may be transmitted to the active control system to be processed with the detected object to execute an automotive vehicle 200 function such as steering, braking or the like.
The method 100 may further include the step of identifying a region of interest 38 for each of the unused frames 32 and corresponding frame 34. The step of identifying a region of interest 38 may be executed by an image processor 24. The region of interest 38 may be determined by the coincidence, identified by the lined section, between the field of views of the first and second cameras 12, 14. The region of interest 38 of the corresponding frame 34 is the same pixel area as the region of interest 38 of the unused frame 32. Accordingly, the image mapping segment 36 may utilize epipolar geometry to determine image information.
The method 100 may further include the step of cropping an area outside of the region of interest 38 is cropped from the corresponding frame 34 and unused frame 32 so as to reduce the amount of data processed. The method 100 may further include the step of removing distortion from the corresponding frame 34 and unused frames 32. The step of cropping and distortion resolution may be executed by the image processor 24 In particular, the image processor 24 may execute a software program that adjusts pixel information to remove distortion. It should be further appreciated that the image processor 24 may identify a region of interest 38, crop and remove distortion from the corresponding and unused frames 32 either before or after the second processing segment 28 matches the resolution of the corresponding frame 34 with the correlating unused frame 32. Preferably, the image processor 24 performs region of interest 38 identification, cropping and distortion removal prior to stereo mapping so as to reduce processing time.
While particular embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, such aspects need not be utilized in combination.