The present invention relates generally to vision systems for vehicles and, more particularly, to vision systems having a plurality of exteriorly facing cameras disposed at a vehicle.
Rear backup cameras and other exterior vision systems are known for use in vehicles. Examples of such systems are described in U.S. Pat. Nos. 7,859,565; 6,611,202; 6,222,447; 5,949,331; 5,670,935; and/or 5,550,677, which are hereby incorporated herein by reference in their entireties. Such systems may display images for viewing by the driver of the vehicle that provide a view exterior of the vehicle. It is known to provide a plurality of cameras at a vehicle, such as a forward facing camera, a rearward facing camera and opposite sideward facing cameras, and to stitch together images captured by the cameras to provide a surround view or top down view for displaying for viewing by a driver of the vehicle.
The present invention provides a means for dynamically calibrating or adjusting the image stitching of the images captured by two or more cameras of a multi-camera vision system of a vehicle, such as when the load or level of the vehicle changes.
The present invention provides a dynamic image stitching system or process that monitors overlapping regions of fields of view of adjacent cameras (such as a side camera and a front or rear camera) and adjusts the image stitching of the captured images to adapt the image stitching when the load of the vehicle changes or when the road level or terrain changes or the like. A display screen displays the stitched substantially seamless images for viewing by a driver of the vehicle when the driver is normally operating the vehicle.
Optionally, the dynamic image stitching system of the present invention may include an image processor that is operable to adjust a stitching algorithm responsive to a determination of a change in at least one characteristic of a feature or object present in an overlapping region of the fields of view of two cameras at the vehicle (such as a side camera and a rear camera or such as a side camera and a front camera or the like). For example, the processor may determine a feature or object present in the overlapping region of the cameras' fields of view, and the processor may compare a characteristic of the feature or object in image data captured by a first camera with the corresponding characteristic of the feature or object in image data captured by a second camera. The processor adjusts the stitching algorithm responsive to a determination of a difference between a characteristic of a feature or object in image data captured by the first camera and the characteristic of the feature or object in image data captured by the second camera. For example, the processor may process captured image data to determine whether the feature as captured by the first camera is misaligned with the feature as captured by the second camera, and the processor may adjust the stitching algorithm responsive to such a determination.
These and other objects, advantages, purposes and features of the present invention will become apparent upon review of the following specification in conjunction with the drawings.
Referring now to the drawings and the illustrative embodiments depicted therein, a vehicle 10 includes an imaging system or vision system 12 that includes at least one exterior facing imaging sensor or camera, such as a rearward facing imaging sensor or camera 14a (and the system may optionally include multiple exterior facing imaging sensors or cameras, such as a forwardly facing camera 14b at the front (or at the windshield) of the vehicle, and a sidewardly/rearwardly facing camera 14c, 14b at respective sides of the vehicle), which captures images exterior of the vehicle, with the camera having a lens for focusing images at or onto an imaging array or imaging plane or imager of the camera (
The dynamic top view vision system of the present invention stitches captured images to form a composite substantially seamless image, and may utilize aspects of the vision systems described in U.S. Pat. Nos. 6,222,447; 5,949,331; 5,760,962; 5,670,935 and/or 5,550,677, and/or U.S. patent application Ser. No. 12/508,840, filed Jul. 24, 2009 (Attorney Docket MAG04 P-1541); Ser. No. 13/848,796, filed Mar. 22, 2013 (Attorney Docket MAG04 P-2034), Ser. No. 13/852,190, filed Mar. 28, 2013 (Attorney Docket MAG04 P-2046), Ser. No. 13/894,870, filed May 15, 2013 (Attorney Docket MAG04 P-2062), and/or Ser. No. 14/046,174, filed Oct. 4, 2013 (Attorney Docket MAG04 P-2158), which are hereby incorporated herein by reference in their entireties.
The panoramic or surround view or top view or bird's eye view vision system of the present invention provides a stitched image that is derived from images or image data captured by a plurality of cameras at the vehicle (such as the forward facing camera, the rearward facing camera, the left side camera and the right side camera). In order to provide a substantially seamless stitched image as displayed on a video display screen viewable by the driver when normally operating the equipped vehicle, respective adjoining image portions of the captured images displayed as a composite image need to be properly aligned, so that, for example, a road marking or parking line or similar feature or characteristic that is at least partially captured by one of the cameras and thus is displayed in its respective image portion, and that is also at least partially captured by a second one of the cameras and thus is displayed in its respective image portion, is properly aligned and displayed on the video screen as it should appear when the cameras are properly calibrated and as it would appear in real life on the ground at or near the vehicle. By means of the present invention, misalignments, discontinuities and improperly joined lines in the stitched composite image that may be caused by the likes of vehicle loading and/or road surface out-of-flatness or the like, can be mitigated and preferably at least substantially eliminated.
Uneven road conditions may make the stitched image look broken and misaligned even with properly calibrated cameras. For example, one or more wheels of the vehicle may be raised higher than the other wheels of the vehicle due to the road camber or tilt. Also, a change in vehicle dynamics that differ from the vehicle dynamics at the time of the initial multi-camera system calibration (when the cameras and vision system were initially calibrated, typically at the automobile manufacturing plant itself) may make the stitched/composite displayed image appear misaligned at the stitching edges/joints. Such a change in vehicle dynamics may be due to uneven or changed load distribution at the vehicle, such as a heavy load added at one side or at the rear of the vehicle or at the top of the vehicle causing the vehicle to tilt towards one side or rearward, or due to uneven passenger loading or the like or uneven vehicle suspension performance or the like. Thus, the present invention provides a dynamic multi-camera top view or bird's eye view or surround view system, which is operable to correct for such run-time/real-time errors or misalignments in the stitched images that would otherwise occur, and is operable to so do on an image-by-image or frame-by-frame basis in real time while the vehicle is being normally operated.
As described in U.S. patent application Ser. No. 13/847,815, filed Mar. 20, 2013 (Attorney Docket MAG04 P-2030), which is hereby incorporated herein by reference in its entirety, a multi camera image processing system may be operable to detect the elevation of objects by its sheering effect when the object is seen in an overlapping region by at least two of the vision system fish eye lens cameras at different angles. The dynamic multi-camera vision system of the present invention also uses the overlapping regions of the fields of view of the respective cameras of the system, such as, for example, the overlapping fields of view of a driver-side camera at an exterior sideview mirror of the vehicle and a front camera (with the overlapping region being, for example, at the front left or driver-side corner of the vehicle). The system is operable to process image data representative of overlapping regions in the adjacent camera fields of view to detect corresponding features. When such corresponding features are detected or determined, the system may match the features.
For example, and with reference to
The system of the present invention is operable to generate affine transforms or other suitable transforms that minimize the error in feature mapping from different views. Such an affine transformation or affine map is a transformation that typically preserves straight lines (such that all points lying on a line initially still lie on a line after transformation) and that typically preserves ratios of distances between points lying on a straight line and that typically preserves sets of parallel lines so that they remain generally parallel after transformation.
The system of the present invention is further operable to apply the transforms to correct the final stitched image. For example, the system may determine when there is a change in the characteristics detected or determined in the overlapping regions of the fields of view and may adapt or adjust the image stitching algorithm or process to accommodate for the detected or determined change. Because the system is a dynamic system, the system is operable to re-generate the transforms for every few frames to correct for dynamic conditions (such as changes in the road conditions or changes in the vehicle loading or the like). The present invention thus is operable to adapt the stitching of the images after the cameras are calibrated to adapt the system and accommodate changes in the vehicle and/or road surface that may otherwise adversely affect the quality or seamlessness of the stitched images. The present invention thus provides a vision system that maintains a substantially seamless stitched image for displaying for view by the driver of the vehicle through various driving and/or loading conditions of the vehicle.
The system thus determines differences in features present in overlapping image data captured by a front (or rear) camera and in overlapping image data captured by a side camera, and may adjust the image stitching algorithm accordingly, such that the displayed stitched image provides an enhanced display (such as by assuring that image data captured by the front camera and image data captured by the side camera appropriately align in the display viewed by the driver of the equipped vehicle so that there is no discontinuity or incongruity in the image being displayed). A feature can be any one of a curb at the side of the road being traveled by the equipped vehicle, a lane boundary marking at the side of the road being traveled by the equipped vehicle, an edge region of the road being traveled by the equipped vehicle, a mailbox being approached and passed by the equipped vehicle, a pole or post or pillar at the side of the road being traveled by the equipped vehicle, and/or the like.
For example, while traveling forward on a road, the equipped vehicle may approach a mailbox located at or near a curb to the left of the equipped vehicle. The vehicle is equipped with a forward facing camera mounted at the front of the vehicle and a side-mounted camera mounted at the left or driver-side exterior rearview mirror of the vehicle. As the mailbox is being approached by the vehicle, it is first viewed/imaged by the front or forward facing camera but is not yet viewed/imaged by the side mounted camera. As the vehicle approaches closer to the mailbox, the mailbox is viewed/imaged by both the forward facing camera and the side-mounted camera, and is located at a region or area where the fields of view of both cameras overlap and encompass the mailbox. In this field of view overlap region, the same object or feature (for example, the mailbox) is commonly viewed by the forward facing camera and the side mounted camera. However, the front camera's view/image of the object or feature will be different from that of the side camera's view/image of the object or feature. In accordance with the present invention, this difference is determined by the image processor and the image stitching algorithm is adjusted according to the determined difference.
For example, if the mailbox is located in the overlap region of two cameras (such as at or near or along a stitching line or region of the captured images), the stitching algorithm is adjusted so that the mailbox images (as captured by, for example, both the front camera and the side camera) generally correspond to one another to provide a generally smooth transition of the mailbox image as the mailbox display image moves or transitions from the front camera image to the overlapping region and to the side camera image. For example, the image stitching algorithm may be adjusted to adjust an orientation of the mailbox so that the generally vertical post or structure of the mailbox in images derived from the image data captured by the front camera is generally aligned with the generally vertical post or structure of the mailbox in images derived from the image data captured by the side camera. The system thus accommodates for misalignment of characteristics of features or objects between the images of the front camera and side camera (or the rear camera and side camera).
Therefore, the present invention provides a dynamic process for automatically stitching images captured by the cameras of a multi-camera vision system of a vehicle, such as for a top-down or bird's eye view display for viewing by the driver of the vehicle. For example, the present invention provides a system that adjusts the stitching of adjacent images captured by the front, rear and driver side and passenger side cameras for a surround view or top-down view vision system of a vehicle.
The present invention utilizes the fact that an object that is located in the overlapping region of the fields of view of two of the cameras of the multi-camera system (such as, for example, a driver-side camera and a rear camera of the vehicle) is commonly imaged by that side camera and that rear camera. However, because of the different and known different placement or positions of the cameras at the equipped vehicle, the object will be viewed differently by the respective first and second camera. Such differences are accommodated for and are factored in via the initial factory calibration so that the system displays the object appropriately and correctly in the stitched composite image. However, if, for example, during dynamic loading of the vehicle, these respective views differ from what should occur in an unloaded or unperturbed ideally calibrated scenario, this determined change or difference can be utilized using image processing/algorithmic techniques to adjust the image stitching to compensate for the dynamic loading of the vehicle in real time. Thus, utilizing the present invention, the stitching of the captured images (to provide or generate the stitched composite image) may be adapted for vehicle load changes and level changes and the like in real time and during normal operation of the vehicle.
Optionally, when the vehicle is in motion (such as driving forwardly or rearwardly), the ground level calibration may be checked permanently or episodically or from time to time, or preferably at times when the system may have unused processing resources. As shown in
The system of the present invention may track objects (such as objects 30 and tracked objects 31 shown in
When there is an offset 33 or a different angle, the side camera will be off by a different angle which points to mis-calibration of the cameras relative to each other (in roll, yaw, pitch angle or transversal displacement) or the vehicle itself is skewed. Typically, both cases can be separated or distinguished by the fact that skewing of the vehicle may happen temporarily while a camera displacement stays in all conditions. Preferably, the system of the present invention may be controlled in a way that first the camera alignment relative to another camera is calibrated and from then on skewing of the vehicle may be assumed when object path inconsistencies appear. Another decision criteria may be that object path inconsistencies may appear (in major) at all four cameras at the same time (such as shown in
Additionally or optionally, the system may also take an input from an environmental sensor 28 (beside the cameras) into account. If not done by the vision computing system alone, the system may utilize distance measuring sensors, such as, for example, ultrasound (exemplary case in
For example, and as shown in
As an additional or alternative way to detect that the vehicle is not fully leveled horizontally (rolled in x axis) and/or has a pitch angle (in y axis) or the like, the system may utilize the compression ratio of the shock absorber system or suspension system of all wheels shock absorbers of the vehicle. Nowadays vehicles usually have sensors to detect the compression rate and usually this signal is provided via the vehicle's CAN or vehicle bus network, which makes it possible that the vision system is able to process this information. By such processing of suspension information (for the individual wheels of the vehicle) the system may determine uneven suspension of the corner regions of the vehicle and may correct the virtual (assumed) ground plane for the virtual top view vision projection in accordance with the determined ratio that the vehicle is rolled or pitched (skewed).
As a preferred embodiment, the system of the present invention may determine the ground level calibration roughly on hand of the shock absorber sensor or suspension data (via CAN) and may use the optical calibration referred to above as a successive fine tuning procedure, which may use the optional range sensor data in addition or in backup.
It is desirable for vision overlays to look quite artificial when these become mapped over a real scene view captured by automotive cameras. Earlier filings, such as International Publication No. WO 2013/086249, which is hereby incorporated herein by reference in its entirety, use visual effects such as ‘motion blurring’ and/or partially transparency in dependence of the viewing distance and direction of the virtual camera to help this fact. Nevertheless there is room for improvement.
As soon as the 3D shape or topography of the world within a displayed scene (in an automotive vision system) is known (using a mono camera and 3D recognition by ‘structure from motion’ or ‘structure from focal length’ or ‘depths estimation from focal blur’ (known from known TV 2D to 3D conversion algorithms (TP-Vision in Philips TV) or by using stereo cameras (including infrared) or using a mono cam era and additionally a second or non-vision-based sensor system, such as LIDAR, RADAR, TOF or the like), especially the ground surface structure the car is resting on, will be renderable. The overlays of the present invention may be mapped or placed (or calculated or controlled or adjusted) responsive to the world's topographical structure. Such overlays may utilize aspects of the overlays described in International Publication No. WO/2009036176, which is hereby incorporated herein by reference in its entirety.
Overlays related to the subject or equipped or host vehicle and overlays dedicated to other vehicles, infrastructure or points of interest (POI) may be mapped onto the structures' topography as like projected or dropped from over top. By that the overlays cover the mapping structure folding around all ups and downs which strengthens the three dimensional impression of the scenery (see
The overlays may have a three dimensional structure which top elevation may be responsive to the ground plane (virtually). By that the top of the overlay repeats the topography of its ground plane (See
The two and three dimensional overlays may be fully or partially transparent for preventing anything important in the scene is hidden by an overlay (and not detected or as important classified). See
With reference now to
Not shown in the drawings is that when the overlays infringe (take the same space) another (real or virtual) object, the overlays may appear in a faded color or in dot or dashed or phantom lines. Optionally, the objects may pile on top of each other (when running across a wall for example). Optionally, the overlays may shrink in size or step slightly to the side when logically possible. For example, an inserted ‘Point of Interest Label’ may give way to a parking overlay which position is systematically crucial.
It is also desirable to gather and provide context depending labeling of information in vehicle vision systems. Storing a lot of information on board the vehicle takes a lot of storage space and it may become outdated comparably fast. Solutions are known to provide useful information to the car aid systems and to the driver via wireless radio channels from and to a dedicated server or from the world wide web, such as by utilizing aspects of the systems described in U.S. patent application Ser. No. 13/660,306, filed Oct. 25, 2012 (Attorney Docket MAG04 P-1950), which is hereby incorporated herein by reference in its entirety. New standards are coming up as like HTML5. This is for providing displaying contents. To maintain the displaying contents up to date adds costs to the system. These costs may be fully or partially borne by third party providers. These may finance their services by adding commercial contents to the ‘useful contents’. For example, these providers may employ a service for ‘points of interest’ (POI) which may come up as virtual labels added to real camera taken scenes of an urban street or environment. The providers may collect money from stores and the company or store logos or labels may be presented in a preferred or selected manner (such as in the foreground, brighter, bigger, more exposed, blinking, or the like).
The vision system of the present invention may employ a spam filter for incoming foreign viewing contents. The spam filter properties may be configurable. The configuration may come from third parties or may be done by the vision system provider or by the vehicle OEM or by the vehicle owner itself (over an HMI) or from a remote device or system (such as, for example, a cell phone app or the like). The spam filter properties may be context dependent (situational, position dependent, vehicle state dependent, vehicle occupation or occupant dependent, driving state dependent). For example there may be contents preferred by the adult occupants, while other or additional contents may become displayed only when a teenager is also on board the vehicle. For example, the teenager may love fast food restaurants while the parents prefer exquisite restaurants. Thus, the vision display system may show POIs of all types of restaurants when the children are in the vehicle with their parents, but will show just some exquisite flagged restaurants when the parents are alone in the vehicle. The “flagging” may come from the vision system provider, by the vehicle OEM or by the vehicle owner itself (over an HMI) or from remote (such as, for example, a cell phone app). Besides that all other annoying content may be optionally suppressible by a spam filter.
Optionally, there may be different spam filter property data sets that the vehicle users are able to store in the vehicle data storage on their cell phone or remote. It may be possible that the spam filter properties (and all other vision system settings) may stay with the driver which ever vehicle he or she uses. The data set may be downloaded from a remote location (server) via any data channel or transferred by his/her cell phone.
The vision system may collect POIs by itself such as, for example, by employing an object detection algorithm on known labels or by requesting the background content according to detected maybe unknown labels. The found POIs may be distributed in a database common to all or to certified users.
Optionally, the vision system (utilizing the forward facing camera and a rearward facing camera and other cameras disposed at the vehicle with exterior fields of view) may be part of or may provide a display of a top-down view or birds-eye view system of the vehicle or a surround view at the vehicle, such as by utilizing aspects of the vision systems described in International Publication Nos. WO 2010/099416; WO 2011/028686; WO 2012/075250; WO 2013/019795; WO 2012/075250; WO 2012/145822; WO 2013/081985; WO 2013/086249; and/or WO 2013/109869, and/or U.S. patent application Ser. No. 13/333,337, filed Dec. 21, 2011 (Attorney Docket DON01 P-1797), which are hereby incorporated herein by reference in their entireties.
The system includes an image processor operable to process image data captured by the camera or cameras, such as for detecting objects or other vehicles or pedestrians or the like in the field of view of one or more of the cameras. For example, the image processor may comprise an EyeQ2 or EyeQ3 image processing chip available from Mobileye Vision Technologies Ltd. of Jerusalem, Israel, and may include object detection software (such as the types described in U.S. Pat. Nos. 7,855,755; 7,720,580; and/or 7,038,577, which are hereby incorporated herein by reference in their entireties), and may analyze image data to detect vehicles and/or other objects. Responsive to such image processing, and when an object or other vehicle is detected, the system may generate an alert to the driver of the vehicle and/or may generate an overlay at the displayed image to highlight or enhance display of the detected object or vehicle, in order to enhance the driver's awareness of the detected object or vehicle or hazardous condition during a driving maneuver of the equipped vehicle.
The vehicle may include any type of sensor or sensors, such as imaging sensors or radar sensors or lidar sensors or ladar sensors or ultrasonic sensors or the like. The imaging sensor or camera may capture image data for image processing and may comprise any suitable camera or sensing device, such as, for example, an array of a plurality of photosensor elements arranged in at least 640 columns and 480 rows (preferably a megapixel imaging array or the like), with a respective lens focusing images onto respective portions of the array. The photosensor array may comprise a plurality of photosensor elements arranged in a photosensor array having rows and columns. The logic and control circuit of the imaging sensor may function in any known manner, and the image processing and algorithmic processing may comprise any suitable means for processing the images and/or image data.
For example, the vision system and/or processing and/or camera and/or circuitry may utilize aspects described in U.S. Pat. Nos. 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331; 6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935; 6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229; 7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287; 5,929,786 and/or 5,786,772, and/or International Publication Nos. WO 2011/028686; WO 2010/099416; WO 2012/061567; WO 2012/068331; WO 2012/075250; WO 2012/103193; WO 2012/0116043; WO 2012/0145313; WO 2012/0145501; WO 2012/145818; WO 2012/145822; WO 2012/158167; WO 2012/075250; WO 2012/0116043; WO 2012/0145501; WO 2012/154919; WO 2013/019707; WO 2013/016409; WO 2013/019795; WO 2013/067083; WO 2013/070539; WO 2013/043661; WO 2013/048994; WO 2013/063014, WO 2013/081984; WO 2013/081985; WO 2013/074604; WO 2013/086249; WO 2013/103548; WO 2013/109869; WO 2013/123161; WO 2013/126715; WO 2013/043661 and/or WO 2013/158592 and/or U.S. patent application Ser. No. 14/163,325, filed Jan. 24, 2014 (Attorney Docket No. MAG04 P-2216); Ser. No. 14/159,772, filed Jan. 21, 2014 (Attorney Docket MAG04 P2215); Ser. No. 14/107,624, filed Dec. 16, 2013 (Attorney Docket MAG04 P-2206); Ser. No. 14/102,981, filed Dec. 11, 2013 (Attorney Docket MAG04 P-2196); Ser. No. 14/102,980, filed Dec. 11, 2013 (Attorney Docket MAG04 P-2195); Ser. No. 14/098,817, filed Dec. 6, 2013 (Attorney Docket MAG04 P-2193); Ser. No. 14/097,581, filed Dec. 5, 2013 (Attorney Docket MAG04 P-2192); Ser. No. 14/093,981, filed Dec. 2, 2013 (Attorney Docket MAG04 P-2197); Ser. No. 14/093,980, filed Dec. 2, 2013 (Attorney Docket MAG04 P-2191); Ser. No. 14/082,573, filed Nov. 18, 2013 (Attorney Docket MAG04 P-2183); Ser. No. 14/082,574, filed Nov. 18, 2013 (Attorney Docket MAG04 P-2184); Ser. No. 14/082,575, filed Nov. 18, 2013 (Attorney Docket MAG04 P-2185); Ser. No. 14/082,577, filed Nov. 18, 2013 (Attorney Docket MAG04 P-2203); Ser. No. 14/071,086, filed Nov. 4, 2013 (Attorney Docket MAG04 P2208); Ser. No. 14/076,524, filed Nov. 11, 2013 (Attorney Docket MAG04 P-2209); Ser. No. 14/052,945, filed Oct. 14, 2013 (Attorney Docket MAG04 P-2165); Ser. No. 14/046,174, filed Oct. 4, 2013 (Attorney Docket MAG04 P-2158); Ser. No. 14/016,790, filed Oct. 3, 2013 (Attorney Docket MAG04 P-2139); Ser. No. 14/036,723, filed Sep. 25, 2013 (Attorney Docket MAG04 P-2148); Ser. No. 14/016,790, filed Sep. 3, 2013 (Attorney Docket MAG04 P-2139); Ser. No. 14/001,272, filed Aug. 23, 2013 (Attorney Docket MAG04 P-1824); Ser. No. 13/970,868, filed Aug. 20, 2013 (Attorney Docket MAG04 P-2131); Ser. No. 13/964,134, filed Aug. 12, 2013 (Attorney Docket MAG04 P-2123); Ser. No. 13/942,758, filed Jul. 16, 2013 (Attorney Docket MAG04 P-2127); Ser. No. 13/942,753, filed Jul. 16, 2013 (Attorney Docket MAG04 P-2112); Ser. No. 13/927,680, filed Jun. 26, 2013 (Attorney Docket MAG04 P-2091); Ser. No. 13/916,051, filed Jun. 12, 2013 (Attorney Docket MAG04 P-2081); Ser. No. 13/894,870, filed May 15, 2013 (Attorney Docket MAG04 P-2062); Ser. No. 13/887,724, filed May 6, 2013 (Attorney Docket MAG04 P-2072); Ser. No. 13/852,190, filed Mar. 28, 2013 (Attorney Docket MAG04 P2046); Ser. No. 13/851,378, filed Mar. 27, 2013 (Attorney Docket MAG04 P-2036); Ser. No. 13/848,796, filed Mar. 22, 2012 (Attorney Docket MAG04 P-2034); Ser. No. 13/847,815, filed Mar. 20, 2013 (Attorney Docket MAG04 P-2030); Ser. No. 13/800,697, filed Mar. 13, 2013 (Attorney Docket MAG04 P-2060); Ser. No. 13/785,099, filed Mar. 5, 2013 (Attorney Docket MAG04 P-2017); Ser. No. 13/779,881, filed Feb. 28, 2013 (Attorney Docket MAG04 P-2028); Ser. No. 13/774,317, filed Feb. 22, 2013 (Attorney Docket MAG04 P-2015); Ser. No. 13/774,315, filed Feb. 22, 2013 (Attorney Docket MAG04 P-2013); Ser. No. 13/681,963, filed Nov. 20, 2012 (Attorney Docket MAG04 P-1983); Ser. No. 13/660,306, filed Oct. 25, 2012 (Attorney Docket MAG04 P-1950); Ser. No. 13/653,577, filed Oct. 17, 2012 (Attorney Docket MAG04 P-1948); and/or Ser. No. 13/534,657, filed Jun. 27, 2012 (Attorney Docket MAG04 P-1892), and/or U.S. provisional applications, Ser. 61/931,811, filed Jan. 27, 2014; Ser. No. 61/919,129, filed Dec. 20, 2013; Ser. No. 61/919,130, filed Dec. 20, 2013; Ser. No. 61/919,131, filed Dec. 20, 2013; Ser. No. 61/919,147, filed Dec. 20, 2013; Ser. No. 61/919,138, filed Dec. 20, 2013, Ser. No. 61/919,133, filed Dec. 20, 2013; Ser. No. 61/918,290, filed Dec. 19, 2013; Ser. No. 61/915,218, filed Dec. 12, 2013; Ser. No. 61/912,146, filed Dec. 5, 2013; Ser. No. 61/911, 666, filed Dec. 4, 2013; Ser. No. 61/911,665, filed Dec. 4, 2013; Ser. No. 61/905,461, filed Nov. 18, 2013; Ser. No. 61/905,462, filed Nov. 18, 2013; Ser. No. 61/901,127, filed Nov. 7, 2013; Ser. No. 61/895,610, filed Oct. 25, 2013; Ser. No. 61/895,609, filed Oct. 25, 2013; Ser. No. 61/893,489, filed Oct. 21, 2013; Ser. No. 61/886,883, filed Oct. 4, 2013; Ser. No. 61/879,837, filed Sep. 19, 2013; Ser. No. 61/879,835, filed Sep. 19, 2013; Ser. No. 61/878,877, filed Sep. 17, 2013; Ser. No. 61/875,351, filed Sep. 9, 2013; Ser. No. 61/869,195, filed. Aug. 23, 2013; Ser. No. 61/864,835, filed Aug. 12, 2013; Ser. No. 61/864,836, filed Aug. 12, 2013; Ser. No. 61/864,837, filed Aug. 12, 2013; Ser. No. 61/864,838, filed Aug. 12, 2013; Ser. No. 61/856,843, filed Jul. 22, 2013, Ser. No. 61/845,061, filed Jul. 11, 2013; Ser. No. 61/844,630, filed Jul. 10, 2013; Ser. No. 61/844,173, filed Jul. 9, 2013; Ser. No. 61/844,171, filed Jul. 9, 2013; Ser. No. 61/842,644, filed Jul. 3, 2013; Ser. No. 61/840,542, filed Jun. 28, 2013; Ser. No. 61/838,619, filed Jun. 24, 2013; Ser. No. 61/838,621, filed Jun. 24, 2013; Ser. No. 61/837,955, filed Jun. 21, 2013; Ser. No. 61/836,900, filed Jun. 19, 2013; Ser. No. 61/836,380, filed Jun. 18, 2013; Ser. No. 61/834,129, filed Jun. 12, 2013; Ser. No. 61/833,080, filed Jun. 10, 2013; Ser. No. 61/830,375, filed Jun. 3, 2013; Ser. No. 61/830,377, filed Jun. 3, 2013; Ser. No. 61/825,752, filed May 21, 2013; Ser. No. 61/825,753, filed May 21, 2013; Ser. No. 61/823,648, filed May 15, 2013; Ser. No. 61/823,644, filed May 15, 2013; Ser. No. 61/821,922, filed May 10, 2013; Ser. No. 61/819,835, filed May 6, 2013; Ser. No. 61/819,033, filed May 3, 2013; Ser. No. 61/816,956, filed Apr. 29, 2013; Ser. No. 61/815,044, filed Apr. 23, 2013; Ser. No. 61/814,533, filed Apr. 22, 2013; Ser. No. 61/813,361, filed Apr. 18, 2013; Ser. No. 61/810,407, filed Apr. 10, 2013; Ser. No. 61/808,930, filed Apr. 5, 2013; Ser. No. 61/806,674, filed Mar. 29, 2013; Ser. No. 61/793,592, filed Mar. 15, 2013; Ser. No. 61/772,015, filed Mar. 4, 2013; Ser. No. 61/772,014, filed Mar. 4, 2013; Ser. No. 61/770,051, filed Feb. 27, 2013; Ser. No. 61/766,883, filed Feb. 20, 2013; Ser. No. 61/760,366, filed Feb. 4, 2013, and/or Ser. No. 61/760,364, filed Feb. 4, 2013, which are all hereby incorporated herein by reference in their entireties. The system may communicate with other communication systems via any suitable means, such as by utilizing aspects of the systems described in International Publication Nos. WO/2010/144900; WO 2013/043661 and/or WO 2013/081985, and/or U.S. patent application Ser. No. 13/202,005, filed Aug. 17, 2011 (Attorney Docket MAG04 P-1595), which are hereby incorporated herein by reference in their entireties.
The imaging device and control and image processor and any associated illumination source, if applicable, may comprise any suitable components, and may utilize aspects of the cameras and vision systems described in U.S. Pat. Nos. 5,550,677; 5,877,897; 6,498,620; 5,670,935; 5,796,094; 6,396,397; 6,806,452; 6,690,268; 7,005,974; 7,937,667; 7,123,168; 7,004,606; 6,946,978; 7,038,577; 6,353,392; 6,320,176; 6,313,454; and 6,824,281, and/or International Publication Nos. WO 2010/099416; WO 2011/028686; and/or WO 2013/016409, and/or U.S. Pat. Publication No. US 2010-0020170, and/or U.S. patent application Ser. No. 13/534,657, filed Jun. 27, 2012 (Attorney Docket MAG04 P-1892), which are all hereby incorporated herein by reference in their entireties. The camera or cameras may comprise any suitable cameras or imaging sensors or camera modules, and may utilize aspects of the cameras or sensors described in U.S. Pat. Publication No. US-2009-0244361 and/or U.S. patent application Ser. No. 13/260,400, filed Sep. 26, 2011 (Attorney Docket MAG04 P-1757), and/or U.S. Pat. Nos. 7,965,336 and/or 7,480,149, which are hereby incorporated herein by reference in their entireties. The imaging array sensor may comprise any suitable sensor, and may utilize various imaging sensors or imaging array sensors or cameras or the like, such as a CMOS imaging array sensor, a CCD sensor or other sensors or the like, such as the types described in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962; 5,715,093; 5,877,897; 6,922,292; 6,757,109; 6,717,610; 6,590,719; 6,201,642; 6,498,620; 5,796,094; 6,097,023; 6,320,176; 6,559,435; 6,831,261; 6,806,452; 6,396,397; 6,822,563; 6,946,978; 7,339,149; 7,038,577; 7,004,606; 7,720,580; and/or 7,965,336, and/or International Publication Nos. WO/2009/036176 and/or WO/2009/046268, which are all hereby incorporated herein by reference in their entireties.
The camera module and circuit chip or board and imaging sensor may be implemented and operated in connection with various vehicular vision-based systems, and/or may be operable utilizing the principles of such other vehicular systems, such as a vehicle headlamp control system, such as the type disclosed in U.S. Pat. Nos. 5,796,094; 6,097,023; 6,320,176; 6,559,435; 6,831,261; 7,004,606; 7,339,149; and/or 7,526,103, which are all hereby incorporated herein by reference in their entireties, a rain sensor, such as the types disclosed in commonly assigned U.S. Pat. Nos. 6,353,392; 6,313,454; 6,320,176; and/or 7,480,149, which are hereby incorporated herein by reference in their entireties, a vehicle vision system, such as a forwardly, sidewardly or rearwardly directed vehicle vision system utilizing principles disclosed in U.S. Pat. Nos. 5,550,677; 5,670,935; 5,760,962; 5,877,897; 5,949,331; 6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563; 6,891,563; 6,946,978; and/or 7,859,565, which are all hereby incorporated herein by reference in their entireties, a trailer hitching aid or tow check system, such as the type disclosed in U.S. Pat. No. 7,005,974, which is hereby incorporated herein by reference in its entirety, a reverse or sideward imaging system, such as for a lane change assistance system or lane departure warning system or for a blind spot or object detection system, such as imaging or detection systems of the types disclosed in U.S. Pat. Nos. 7,720,580; 7,038,577; 5,929,786 and/or 5,786,772, and/or U.S. pat. application Ser. No. 11/239,980, filed Sep. 30, 2005, now U.S. Pat. No. 7,881,496, and/or U.S. provisional applications, Ser. No. 60/628,709, filed Nov. 17, 2004; Ser. No. 60/614,644, filed Sep. 30, 2004; Ser. No. 60/618,686, filed Oct. 14, 2004; Ser. No. 60/638,687, filed Dec. 23, 2004, which are hereby incorporated herein by reference in their entireties, a video device for internal cabin surveillance and/or video telephone function, such as disclosed in U.S. Pat. Nos. 5,760,962; 5,877,897; 6,690,268; and/or 7,370,983, and/or U.S. Publication No. US-2006-0050018, which are hereby incorporated herein by reference in their entireties, a traffic sign recognition system, a system for determining a distance to a leading or trailing vehicle or object, such as a system utilizing the principles disclosed in U.S. Pat. Nos. 6,396,397 and/or 7,123,168, which are hereby incorporated herein by reference in their entireties, and/or the like.
Optionally, the circuit board or chip may include circuitry for the imaging array sensor and or other electronic accessories or features, such as by utilizing compass-on-a-chip or EC driver-on-a-chip technology and aspects such as described in U.S. Pat. No. 7,255,451 and/or U.S. Pat. No. 7,480,149; and/or U.S. patent application Ser. No. 11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 as U.S. Publication No. US-2006-0061008, and/or Ser. No. 12/578,732, filed Oct. 14, 2009 (Attorney Docket DON01 P-1564), which are hereby incorporated herein by reference in their entireties.
Optionally, the vision system may include a display for displaying images captured by one or more of the imaging sensors for viewing by the driver of the vehicle while the driver is normally operating the vehicle. Optionally, for example, the vision system may include a video display device disposed at or in the interior rearview mirror assembly of the vehicle, such as by utilizing aspects of the video mirror display systems described in U.S. Pat. No. 6,690,268 and/or U.S. patent application Ser. No. 13/333,337, filed Dec. 21, 2011 (Attorney Docket DON01 P-1797), which are hereby incorporated herein by reference in their entireties. The video mirror display may comprise any suitable devices and systems and optionally may utilize aspects of the compass display systems described in U.S. Pat. Nos. 7,370,983; 7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551; 5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,677,851; 5,708,410; 5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,508; 6,222,460; 6,513,252; and/or 6,642,851, and/or European patent application, published Oct. 11, 2000 under Publication No. EP 0 1043566, and/or U.S. patent application Ser. No. 11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 as U.S. Publication No. US-2006-0061008, which are all hereby incorporated herein by reference in their entireties. Optionally, the video mirror display screen or device may be operable to display images captured by a rearward viewing camera of the vehicle during a reversing maneuver of the vehicle (such as responsive to the vehicle gear actuator being placed in a reverse gear position or the like) to assist the driver in backing up the vehicle, and optionally may be operable to display the compass heading or directional heading character or icon when the vehicle is not undertaking a reversing maneuver, such as when the vehicle is being driven in a forward direction along a road (such as by utilizing aspects of the display system described in PCT Application No. PCT/US2011/056295, filed Oct. 14, 2011 and published Apr. 19, 2012 as International Publication No. WO 2012/051500, which is hereby incorporated herein by reference in its entirety).
Optionally, a video mirror display may be disposed rearward of and behind the reflective element assembly and may comprise a display such as the types disclosed in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755; 7,626,749; 7,581,859; 7,446,650; 7,370,983; 7,338,177; 7,274,501; 7,255,451; 7,195,381; 7,184,190; 5,668,663; 5,724,187 and/or 6,690,268, and/or in U.S. patent application Ser. No. 12/091,525, filed Apr. 25, 2008, now U.S. Pat. No. 7,855,755; Ser. No. 11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 as U.S. Publication No. US-2006-0061008; and/or Ser. No. 10/538,724, filed Jun. 13, 2005 and published Mar. 9, 2006 as U.S. Publication No. US-2006-0050018, which are all hereby incorporated herein by reference in their entireties. The display is viewable through the reflective element when the display is activated to display information. The display element may be any type of display element, such as a vacuum fluorescent (VF) display element, a light emitting diode (LED) display element, such as an organic light emitting diode (OLED) or an inorganic light emitting diode, an electroluminescent (EL) display element, a liquid crystal display (LCD) element, a video screen display element or backlit thin film transistor (TFT) display element or the like, and may be operable to display various information (as discrete characters, icons or the like, or in a multi-pixel manner) to the driver of the vehicle, such as passenger side inflatable restraint (PSIR) information, tire pressure status, and/or the like. The mirror assembly and/or display may utilize aspects described in U.S. Pat. Nos. 7,184,190; 7,255,451; 7,446,924 and/or 7,338,177, which are all hereby incorporated herein by reference in their entireties. The thicknesses and materials of the coatings on the substrates of the reflective element may be selected to provide a desired color or tint to the mirror reflective element, such as a blue colored reflector, such as is known in the art and such as described in U.S. Pat. Nos. 5,910,854; 6,420,036; and/or 7,274,501, which are hereby incorporated herein by reference in their entireties.
Changes and modifications in the specifically described embodiments may be carried out without departing from the principles of the present invention, which is intended to be limited only by the scope of the appended claims as interpreted according to the principles of patent law.
The present application claims the filing benefits of U.S. provisional applications, Ser. No. 61/793,592, filed Mar. 15, 2013, and Ser. No. 61/770,051, filed Feb. 27, 2013, which are hereby incorporated herein by reference in their entireties.
Number | Date | Country | |
---|---|---|---|
61793592 | Mar 2013 | US | |
61770051 | Feb 2013 | US |