Patent Application
20110010041
The invention is related to a flexible method for hazard detection and information implemented in a vehicle with at least one side and rear area sensing device, area interpretation device and a possibility to display the information and/or warning signals for the driver.
From DE 44 10 620 A1 is known a monitoring device for the driver and or passenger side of vehicles. The monitoring device comprises a sensor in the vehicle's exterior mirror for monitoring the blind spot region. The sensor, an ultrasonic or infrared sensor, is connected to a control unit that causes a visual signal to light up in the exterior mirror in the event that an object is detected in the blind spot in order to warn the driver. Object identification or predictive interpretation of motion is not possible here. There is a lot of prior art published related to single and stand alone solution for an assistant system in a vehicle which provides some additional benefit to driver and assist him in special actions.
The plurality of different assist systems that often uses radar or ultrasonic sensors for their purpose makes is complicated for the car manufacturer to decide which assistant system should be installed. To install different independent systems reduces the effectiveness of the assistant function and the user will be overwhelmed by information flow.
The present invention is thus intended to solve the problem of developing a flexible software for an automotive hazard detection and information system with at least one area sensing device that automatically detects present an impending hazardous situations or environmental situations for information and induces the driver at least assess the situation visually.
The problem is solved by the features of the main claim. To this end the software of the hazard detection system works with one sensor or a sensor group and is structured in modules to allow the flexible activation of modules by OEM and/or the driver. The software runs on a device including at least one analysis and interpretation unit per sensor or sensor group to determine geometry data and motion data of the object or objects sensed. It has at least one display unit for each sensor or sensor group.
With the aid of the sensors affixed to the outside of the vehicle here, moving traffic, etc., for example in the blind spot region to the rear of the exterior mirror or mirrors is detected. By means of an analysis and interpretation unit, the images or contours detected by the sensors provide object characterization with respect to size or type, and the image sequences provide the relative motions of the object or objects observed. From the geometry and motion data, the analysis unit calculates a possible collision or near collision, in the event that the present courses of all objects involved are maintained. In hazardous cases, the driver is warned by visual, acoustic, or tactile means, and if applicable is informed and/or prompted with respect to possible reactions to avert the danger.
Further details of the invention are evident from the dependent claims and the description below of schematically illustrated example embodiments.
Alternatively, the sensor (11) located in the front, viewed in the direction of travel, can be accommodated in the exterior or interior mirror, on the mirror triangle for the outside mirror, on the third side directional signal, or in the grip strip of the driver-side door handle, among other locations. If the sensor (11) is integrated in a mirror, it can be located behind the mirror glass, on the mirror housing, or in the mirror base.
The mirror triangle is a region of the outer vehicle shell. It is generally part of the driver-side or passenger-side door and is located between the doorpost near the A-pillar and the door-side window. The mirror triangle carries and positions the exterior mirror on the driver or passenger door and will carry in future a camera module only to replace a rear view mirror.
As an alternative to placement in the rear light unit (14), the rear sensor (15) can be positioned, for example, in the rear bumper, in the region of the tailgate handle, in the centre auxiliary brake light, in the license plate light, or in a passenger compartment vent integrated in the C-pillar or D-pillar. Within the rear light unit (14), the sensor can be placed in the back-up light, in the turn signal light, in the taillight, in the rear fog light, or in the rear brake light.
The sensors (11, 15) can be digital cameras, range-finding cameras, laser systems or radar systems, for example. Motion sensors and other range measurement systems are also possible. Different sensor types can also be combined in a sensor group. Probably the sensor is an optical sensor that can be combined with other sensor types in a sensor group.
The front sensor (11) has an angle of view of approximately 60 to 80 degrees, with the line delimiting the field of view (13) nearest the vehicle extending along the outer contour of the vehicle body (6); in other words, this delimiting line (13) extends parallel to the direction of travel, for example. The detection and/or analysis range is 10 to 60 meters, for example.
The angle of view of the rear sensor (15) covers approximately 15 to 50 degrees, for example. The detection and/or analysis range extends to 30 to 40 meters.
The optical sensor uses a wide angle optic, preferably a fish-eye optical lens that extend the angle ranges to a much higher extend than discussed above. The wide range view allows a better coverage of areas adjacent to the vehicle and increases the functions that can be implemented using the wide angle data set.
The purpose of the sensors (11, 15) is to sense the surroundings. They are meant to detect objects in motion, for example driving objects (2, 3), which move relative to the vehicle (1) in such a way that a later collision cannot be ruled out if the driver of vehicle (1) does not react by changing the direction of travel (7) or the speed. Through appropriate processing of the sensor data, the direction of motion, speed, and changes therein, are continuously calculated in an interpretation unit and compared with the comparable data for the vehicle (1). From these data are calculated a possible collision point or an encounter that is still collision-free but closer than a minimum distance. Both possibilities are interpreted as a hazardous situation.
Driver reaction assistance is derived from this. From the fact that vehicle direction (7) and speed are maintained, or from a change in one or both that increases a risk of collision, the hazard detection system interprets that the driver of (1) does not perceive the approaching object (2, 3) in the exterior mirror blind spot. In a first phase, the system forces the drive to look in the exterior mirror (10) by means of a lighted or blinking visual signal on or in the vicinity of the exterior mirror (10). Generally, the driver of (1), continuing not to perceive a hazard, will look back over his shoulder on the side facing the appropriate exterior mirror (10), notice the vehicle (2, 3) to the rear, and react appropriately to avert a danger.
Nearly any type of acoustic warning can assist the driver of (1) in this situation.
If the driver of (1) continues to evidence no reaction, in a second phase his attention is drawn to an imminent hazardous situation. In this embodiment the steering wheel and/or the brake or gas pedal serves as an information device. To this end, the steering wheel and/or the relevant pedal is set into a pulsing motion. As a rule, this pulsing motion has no direct effect on steering action or vehicle acceleration. Independently of this, if desired, the brake pressure is increased, for example, in order to shorten the braking response time.
Moreover, it is also possible for the system to activate the hot-air fan at a certain difference between the inside and outside temperatures and/or at or above a certain air humidity level in the interior air, and, by means of the ventilation grille (26), dry the side window, at least in the area of the exterior mirror, in order to improve visibility of the exterior mirror. Active adjustment of the ventilation louvers for optimal hot-air conduction is also possible.
The pulsing or vibrating motion of the steering wheel or at least one of the pedals makes the driver of (1), who has physical contact with at least the accelerator or steering wheel, expressly aware of a general or specific hazardous situation. For example, the vibration of the steering wheel can prepare him through tactile means for the need to avert the hazardous situation by turning the steering wheel. In addition or alternatively, in the case of a hazardous situation that turning the steering wheel would avert, the driver's seat can be palpably tilted in the direction in which the driver should steer. If desired, the seat and/or backrest can also vibrate in the process. In addition, instead of tilting of the seat—for example if it is necessary to steer toward the right—the left side of the driver's buttocks could be raised or the right side could be lowered. The palpable unilateral or alternating lifting can be in the range of millimetres.
The visual signal from the first warning phase is emitted by a light source in the form of an indicator lamp (41-61). Such lamps are shown in
In addition,
In
In vehicles without A-pillar panelling, the blind spot lamp (41) sits directly in a recess worked in the hollow profile of the A-pillar (21).
In
In
As shown in
[The warning lamp (61) shown in
The luminous intensity of the warning lamps (41-61) is adapted to the ambient brightness if desired, i.e. the brighter the environment, the more intensely the warning lamp (41-61) glows. The lamp lens material can be a transparent plastic, glass, or a comparable material. If desired, the lamp lens is simultaneously the body of the light source, e.g. the bulb of the incandescent lamp or the housing of an LED or LED array.
The step to find features for tracking is different for night or day use. In darkness the objects can be tracked by their headlamps, so that a vehicle or a motor bike can be detected and the vector of motion tracked. The software is able to distinguish between one or two headlamps and includes a plausibility check. For example it is checked whether the detected headlamps are not higher than a threshold height over street level so that no other artefacts are tracked in night. A street lamp that is detected would be rejected by the software because of the height over ground level.
In the day mode the step of finding features in the image is different. The applied algorithm uses the vanishing points of structures in the image to define objects and their position in the three dimensions. The objects detected are qualified into danger categories. In the decision stage software decides about a warning signal yes or no, to be presented to vehicle's driver. Basis for the warning is use of object classification and a decision tree that allows to limit warning to only hazardous situation. The bus-data input on this stage provides vehicle parameters that can be involved in the warning decision, as velocity, steering angle, setting of turn signal indicator etc.
A software module is a small self-contained program that carries out a clearly defined task and is intended to operate within a larger program suite.
The abbreviations D and W explain whether the output of a software module is used to be displayed in form of an image on a display or used for a warning indication or for both.
After the step of data reduction the same data set is used as parallel input for all blocks representing software modules of
Auto calibration is a software module that allows the calibration of field of view independent from the installation and the mounting of the optical sensor in the production line.
The software for hazard detection and information system is self-adjusting. It is important to adjust the area of detection of a hazard detection system to achieve a reliable warning situation. For the software is analysing images and deriving the hazardous status from the detected objects the view of the optical sensor must be defined. The very sensitive function of a warning system is sensible to the final optical view of the optical sensor. During normal vehicle mounting the view of a sensor can change slightly. In prior art the view of the camera sensor must be amended after the rear view mirror is attached to the vehicle or the senor is mounted somewhere else at the vehicle to tune the warning system. In details the position of the sensor must be adapted using defined target lines in the production line and amending the position. This increases the cost of installation of the assistant system.
According to the invention the software recalculates the actual position and adapt the field of view to an optimum to initiate the hazardous warning system.
The software module auto calibration is also used to adapt the field of view if the vehicle carries a big load and the position of the cameras sensor changes. The software uses vanishing points of detected features of the image that are calculated to define the view angles versus the vehicle coordinates.
This flexibility is possible because an optical sensor with a wide angle optic has a very broad view and the field of view is shifted by software over the total range of the image recorded.
Another software module is able to detect dirt or fogging on the optical lens.
If both a right-side and left-side sensors are used, both software modules can work together or can be independent of one another. Differences arising here can also be analyzed. This is a big advantage using optical sensor systems. The optical sensors are able to compare data to detect for example any distortion that covers one of the optical lenses. The optical sensors are in some cases hidden by dust or a water droplet. These distortions arise failures in object detection. By comparing the situation on both sides of the vehicle these issues are solved. The software compensates a distortion on a lens and the object detection can successfully detect a hazardous situation again.
Either auto calibration or dirt detection are modules that in background parallel to further software modules.
Another background running module is the shadow detection. This software module derives data that shows light and dark regions in the image data that are correlated to the own vehicle, The shadow detection is important to avoid mis-warnings that occurs from the light-dark-transition and the features that are track in the image therefore.
Blind sport detection module is a function as discussed for
Another software module is detecting traffic sign, which is comparable with other object detections. Also lane sign detection is done in a further software module. The lane detection is then used for two different further software module, the lane departure warning which implements a forward looking view of the optical sensor and and the lane change control.
The data of the optical sensor are further used to rotate and recalculate the image pixel by pixel. These results in an image which give the bird view sight of the vehicle's surrounding.
Another software module used the image for a park assistant function that can be combined with Bird's View function.
A software module for collision control and warning is also available.
The hazard detection system can be designed as a complete module that requires only few vehicle data. The system is realized with a controller connected via a bus system with the vehicle. The controller is either able to calculate all the functions of the software for example the object recognition issues or is connected to another controller that is specially adapted for video calculations in real time. Data exchange with the vehicle can take place over a LIN bus or CAN bus, for example. For adaptation to the standard bus systems the controller includes the LIN- or CAN bus functionality that allows the communication with a master controller in the vehicle. The module consisting of sensor, optic, electrical circuit and at least one controller can be placed directly on or in the rear view mirror housing or elsewhere at the vehicle. The position of the optical sensor is only limited by the field of view and by shading of this field of view by vehicle. Thus it can be permanently attached and also ensure good thermal transfer to the vehicle body. The hazard detection system with a standard controller connected to the vehicle bus system is also adapted to control more than the optical sensor data. The control of further data of devices that are mounted or attached to a rear view mirror is realized. One example is the control of a turn signal indicator.
The software of hazard detection and information system can also be used for the traffic space in front of the vehicle. If desired, the monitoring of the traffic space in front of the vehicle and behind the vehicle can be integrated in one module. Then, for example, pedestrians, traffic signs, special-purpose vehicles such as police, fire trucks, etc., can also be detected with the aid of the optical sensors and operated by the relevant software module.
The warning indication is provided by means of a lamp arranged on the edge of the driver's field of view. The warning then takes the form of flashing of the light, for example. The lamp, for example a light-emitting diode, can be directed at the driver.
The software of hazard detection and information system includes a brightness detection device. Thus it can switch over from a day mode to a night mode, possibly coupled to the on-board clock, and use the appropriate software program based on the mode. Normally the day and night situation is derived from the received data of the optical sensor.
Also, in the event of, e.g., mis-adjustment of the sensors, a warning message can be issued or the software can automatically adjust itself or compensate for a mis-adjustment. It is likewise possible for, e.g., the size and direction of the monitored region to be adjustable or settable by the driver, for example by means of an operating display.
The type of warning message and if applicable the control signal issued by the hazard detection system can be governed by the severity of the danger. For example, they can be dependent on the travel speed of the monitoring vehicle, the travel speed of the monitored vehicle in the hazard zone, the radius of turn, etc.
The sensors (11, 15) can—as already mentioned—include an ordinary commercial camera and ordinary commercial optical lenses. These components are arranged directly behind a window in a housing that forms a module in or at the rear view mirror housing. The electrical components are then designed specifically for the hazard detection and information system. Close to the camera or optical sensor chip an image processing unit is placed to receive data from the sensor and start calculations. The image processor unit is linked to a processor or controller that analyses and interprets the data for further use. All software is either stored in related storages of the units or controller either on one device or spread over several devices.
The individual camera has a field of view of more than to 60 degrees using a wide angle lens. The lenses can have a hydrophilic or hydrophobic coating that is applied as a permanent coating, or is renewed during cleaning.
The module can also be arranged in an area of the door that is subjected to moisture. The module can then be designed with IP 67 protection. In this context, the wiring is in the sealed area. Large-volume, sealed connectors can be eliminated. The lens and the CMOS electronics are then glued into the housing.
The thermal expansions of the different device parts are compensated with a Gore Tex® seal. This achieves, firstly, water tightness preventing the entry of moisture, and secondly prevents the build-up of an overpressure of air in the housing.
The software of the hazard detection system can be customer-specific. It can be programmed by the driver or the service shop, for example. It can be used for other applications in the vehicle in addition to the hazard detection system. For example, an interface to the data network of the vehicle can be provided. Thus, for example, the settings of the sensors (11, 15) can automatically be compensated depending on the loading of the vehicle (1). In addition, various operating states or driver-specific settings can be pre-programmed, for example based on the driver's specific field of view. In this regard, for example, it is possible to consider the seat position of the driver, the individual visual acuity of the driver, the reaction time of the driver, etc.
Modular software structure comprises the possibility to create software adaptable to driver's need or to the actual parameters. A system with optical sensors can be used for several functions, blind spot detection rearwardly, blind spot detection forwardly, lane change detection and assistant, parking assistant, bird's view application, surrounding monitoring. There are additional modules for traffic sign recognition, and for other assistant functions. The module auto calibration and the module dirt detection are always active.
According the acquired package software is activated. The user of a vehicle equipped with this software is able to receive updates of the functionality of the hazardous warning system or to have additional functions installed. In order to protect the sensors (11, 15), they can be equipped with an electromechanically operated cover that is closed when the vehicle (1) is stopped. A cleaning mechanism for the lens, for example a wiper, spray nozzles, etc. is also possible.
The speed of the object (2, 3) relative to that of the vehicle (1) bearing the system may be minimal. Thus even if two vehicles travelling at approximately the same speed should approach one another during a lane change, this can be detected. Conversely, even stopped vehicles can be detected.
The hazard detection system can also warn if there is and/or will be reduced visibility. This can be caused by, e.g., a dirty lens, fog, etc. To this end, the hazard detection system has infrared sensors in addition to sensors (11, 15) that detect objects (2, 3) in the visible spectrum. These cited functions can also be combined in a single sensor (11, 15). The use of a night vision device in the hazard detection system or in combination therewith is also possible.
The CMOS electronics of the signal generator produces a black-and-white image. The image produced can thus have a high pixel density. The invention is not limited to a black and white optical sensor. Colour sensors are used if the image is displayed to give a real view of surrounding.
The visual warning signal in the mirror triangle (25), for example the blind spot lamp (51) designed in the shape of an arrow, is designed such that it elicits a glance at the mirror (10, 18). The warning signal alone thus does not provide complete information about the hazard and does not replace a look in the mirror (10, 18).
The hazard detection system can perform self-diagnostics with regard to its function. Thus, it can regularly receive signals and report back over the data connection line from the vehicle (1). A special diagnostics interface is also possible.
The units can be used equally well for driving on the right or left.
The hazard detection system can monitor and recognize multiple vehicles (2, 3) simultaneously, and warn of possible hazards caused by these vehicles (2, 3). In this regard, for example, a closer vehicle (3) can be assessed as the primary danger and a vehicle (2) that is further away as a lesser danger.
The individual mirror (10, 18) can be attached by its base surface. In this way, it can be largely insensitive to vibrations, shocks, etc. In addition, the sensors (11, 15) can then be integrated in the base plate, for example. Adjustment of the mirror (10, 18) then does not affect the position of the sensors (11, 15). The sensors (11, 15) are then adjusted by means of an adjusting screw on the outside of the mirror (10, 18.
If the sensors are mounted in the pivotable part of the rear view mirror the view of the sensor can be adapted by the software. The software defines the area of view actually and calculated the hazardous situations according the actual views.
The sensors (11, 15) can also be arranged in such locations as on the roof, in the doors, in the rear window, in the trunk lid, etc. Thus the detection regions (12, 16) can have a large overlap.
The hazard detection system can be used in the customary temperature range. Thus, even low temperatures and windows that are partly iced up, do not cause the hazard detection system to fail. In order to minimize the effects of extremely high or extremely low temperatures, the hazard detection system can also have protection against heat and/or cold, a heater, a fan, and/or a defroster for the lens. The individual components of the hazard detection system can also be electrically shielded. In this way, electrical influences on the hazard detection system by other vehicle components or electric and/or magnetic fields in the surroundings of the vehicle (1) can be prevented. In addition, the hazard detection system does not electrically and/or magnetically affect any other vehicle components or the environment.
The components of the hazard detection system can be arranged in a housing that is protected against unauthorized access. Thus, for example, it can be sealed or it can be closed with special screws. The housing can be made of die-cast aluminium with an anodic coating as corrosion-proofing, for example. A hermetic sealed housing helps to protect electrical devices including controller from dust and humidity. The sealed module is also resistant against electromagnetic pulses and does not sent electromagnetic distortion pulses.
If a camera is used as a sensor (11, 15), the camera can include an auto focus. The optics of the camera are then set such that objects in the more remote environment do not affect the sensors (11, 15). The brightness information of the image recorded by the camera can be used to adjust the brightness of the warning lamp (41-61). The warning lamps (41-61) are arranged such that the driver can see them without turning his head, for instance.
In order to install the hazard detection system in a motor vehicle, the standard mirror can be replaced by a mirror (10, 18) that includes components of the hazard detection system.
The images detected by the sensors (11, 15) and the information determined there from can be stored inside the vehicle (1), for example together with the operating parameters of the vehicle. In this way, an accident can be reconstructed after the fact. The data can also be transmitted to a removable storage medium. Even wireless transmission is possible, either in real time or at regular time intervals. The data can also already be compressed and processed at this point.
At least one of the sensors (11, 15) can also be arranged on a vehicle that is not self-propelled, such as a trailer, a semi trailer, etc. The towed vehicle is then connected to the data bus of the towing vehicle, if desired even over a wireless connection.
In the event of trailer operation, the hazard detection system can also sense yawing of the trailer, for example, before the trailer jack-knifes relative to the towing vehicle.
Software of hazard detection and information system includes other systems assisting the driver, such as assistive braking, assistive lane changing, etc. In this regard, software can be adapted to the course of travel. For instance, if the assistive lane changing system detects a lane change, software can monitor the danger zone accordingly. Also, the image from one or more cameras of the assistive lane changing system can be projected on the windshield together with the image from one or more cameras of the system. Then a complete image of the traffic space to the rear is presented on the head-up display. If desired, the mirrors (10, 18) can be omitted entirely in this case. In this context, the software has an interface with the internal data bus by three wires and is connected to the head-up display by a two-wire line.
In hazard situations and on the highway, software system can supply data in order to influence, at least the steering of the vehicle.
The hardware of the system can also contain additional components, such as curb lights, mirror elements with controlled dimming and/or tint, heaters, etc that are directly of via bus system connected to the software The data transmission and the control here can be accomplished through glass fibres, wireless transmission, etc. In addition one module of the software includes an access control system with thermal profile monitoring for keyless access to the vehicle.
The sensors (11, 15) can be arranged separately from the analysis unit. The sensors (11, 15) can be placed at the top or bottom of the mirror (10, 18). They are then connected to the analysis unit, which is located in the door, in the frame, on the inside of the door, the mirror triangle (25), etc.
It is possible to deactivate the software of hazard detection and information system complete or only per modules. Thus the blind spot warning module can be switched off for parking, to enter a garage, in traffic jams, in heavy traffic, etc. but the parking assistant still runs. Switch-on and switch-off can be done automatically or by the driver.
The invention has been described in an illustrative manner. It is to be understood that the terminology, which has been used, is intended to be in the nature of words of description rather than of limitation.
Many modifications and variations of the invention are possible in light of the above teachings. Therefore, within the scope of the appended claims, the invention may be practiced other than as specifically described.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10303578.8 | Jan 2003 | DE | national |
This patent application is a continuation-in-part patent application claiming priority to a United States patent application having application Ser. No. 10/543,910, filed Jun. 22, 2006, claiming priority to a patent application claiming priority to PCT/DE2004/000140, filed Jan. 30, 2004, claiming priority to a German patent application having application number 103 03 578.8, filed Jan. 30, 2003.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10543910 | Jun 2006 | US |
| Child | 12884944 | US |
