Patent Application
20210239485
The present disclosure relates to a system and a method for navigation of a motor vehicle using terrain text recognition.
A vehicle navigation system may be part of integrated vehicle controls or an add-on apparatus used to find direction in the vehicle. Vehicle navigation systems are crucial for the development of vehicle automation, i.e., self-driving cars. Typically, a vehicle navigation system uses a satellite navigation device to obtain its position data, which is then correlated to the vehicle's position relative to a surrounding geographical area. Based on such information, when directions to a specific waypoint are needed, routing to such a destination may be calculated. On-the-fly traffic information may be used to adjust the route.
Current position of a vehicle may be calculated via dead reckoning—by using a previously determined position and advancing that position based upon known or estimated speeds over elapsed time and course. Distance data from sensors attached to the vehicle's drivetrain, e.g., gyroscope and accelerometer, and vehicle mounted radar and optical equipment may be used for greater reliability and to counter global positioning system (GPS) satellite signal loss and/or multipath interference due to urban canyons or tunnels. In urban and suburban settings, locations of landmarks, sights, and various attractions are frequently identified via signs employing textual description or formal name of the point of interest.
A method of vehicle navigation using terrain text recognition includes receiving, via an electronic controller arranged on a vehicle and having access to a map of the terrain, a navigation route through the terrain. The method also includes receiving, via the controller, a signal from a global positioning system (GPS) and using the signal to determine a current position of the vehicle relative to the terrain. The method additionally includes determining, via the electronic controller, a location of a next waypoint on the navigation route and relative to the current position of the vehicle. The method also includes detecting and communicating to the electronic controller, via a sensor arranged on the vehicle an image frame displaying a text indicative of the next waypoint. The method additionally includes correlating, via the electronic controller, the detected text to the next waypoint on the map of the terrain. Furthermore, the method includes setting, via the electronic controller, an in-vehicle alert indicative of the detected text having been correlated to the next waypoint.
The method may also include determining a distance from the current position to the determined location of the next waypoint.
Additionally, the method may include determining whether the distance from the current position to the determined location of the next waypoint is within a threshold distance.
According to the method, setting the in-vehicle alert may be accomplished when the distance from the current position to the determined location of the next waypoint is within the threshold distance.
According to the method, correlating the detected text to the next waypoint on the map of the terrain may include using a trained Neural Network architecture.
The Neural Network architecture may be a unified Neural Network structure configured to recognize the image frame. The unified Neural Network structure may include a fully-convolutional first Neural Network having an image input and at least one layer, and configured to recognize the text. The unified Neural Network structure may also include a convolutional second Neural Network having a text input and at least one layer. In such a structure, an output from the at least one layer of the second Neural Network may be merged with the at least one layer of the first Neural Network. The first and second Neural Networks may be trained together to output a mask score.
According to the method, setting the in-vehicle alert indicative of the detected text having been correlated to the next waypoint on the map of the terrain may include projecting, via a head-up display (HUM a highlight icon representative of the mask score onto a view of the next waypoint.
The method may also include determining a field of vision of an occupant of the vehicle and setting the in-vehicle alert in response to the determined field of vision.
According to the method, determining the field of vision may include detecting an orientation of a vehicle occupant's eyes. In such an embodiment, the in-vehicle alert may include projecting the highlight icon in response to the detected orientation of the vehicle occupant's eyes.
According to the method, setting the in-vehicle alert may include triggering an audible signal when the next waypoint appears in the determined field of vision.
A system for vehicle navigation using terrain text recognition and employing the above-described method is also disclosed.
The above features and advantages, and other features and advantages of the present disclosure, will be readily apparent from the following detailed description of the embodiment(s) and best mode(s) for carrying out the described disclosure when taken in connection with the accompanying drawings and appended claims.
Referring to the drawings, wherein like reference numbers refer to like components,
As shown in
As shown in
Specifically, an optical device may be a laser beam source for a Light Detection and Ranging (LIDAR) system, or a laser light sensor for an adaptive cruise control system or a camera capable of generating video files. In an exemplary embodiment of the system 18, the first vehicle sensor 20 may be a camera and the second vehicle sensor 22 may be a LIDAR. In general, each of the first and second vehicle sensors 20, 22 is configured to detect the immediate surroundings of the terrain 14, including, for example, an object 24 positioned external to the vehicle 10. The object 24 may be a specific point of interest, such as a landmark, a building structure housing a particular business establishment, a road, or an intersection, each identified via a respective sign employing a textual description or a formal name of the subject point of interest.
The data gathering and processing system 18 also includes a programmable electronic controller 26 in communication with the first and second sensors 20, 22. As shown in
The controller 26 may communicate with the respective first and second sensors 20, 22 via a transmission medium, including coaxial cables, copper wire and fiber optics, including the wires in a system bus coupling a specific controller to an individual processor. Memory of the controller 26 may also include a flexible disk, hard disk, magnetic tape, other magnetic medium, a CD-ROM, DVD, other optical medium, etc. Controller 26 may be equipped with a high-speed primary clock, requisite Analog-to-Digital (A/D) and/or Digital-to-Analog (D/A) circuitry, input/output circuitry and devices (I/O), as well as appropriate signal conditioning and/or buffer circuitry. Algorithms required by the controller 26 or accessible thereby may be stored in the controller memory and automatically executed to provide the required functionality. Controller 26 may be configured, i.e., structured and programmed, to receive and process captured raw data signals gathered by the respective first and second sensors 20, 22.
As shown, the electronic controller 26 includes a navigation module 30. Physically, the navigation module 30 may be arranged separately from the controller 26 or be housed therein. The navigation module 30 includes a map 32 of the geographical area with the terrain 14 stored within its memory, and is generally configured to establish navigation routes for guidance of the vehicle 10 through the terrain 14. The navigation module 30 is configured to determine a navigation route 34 to a particular destination 36 through the terrain 14, such as following a request for determination of the subject route by an operator of the vehicle 10. The controller 26 is specifically configured to access the navigation route 34 in the navigation module 30 or receive the navigation route therefrom. The navigation module 30 is generally configured to output the determined navigation route 34 and display the route on a navigation screen 39 (shown in
As defined herein, the data gathering and processing system 18 also includes a global positioning system (GPS) 38 having earth-orbiting satellites in communication with the navigation module 30. The controller 26 is configured to receive from the GPS 38, such as via the navigation module 30, signal(s) 38A indicative of a current position of the GPS satellite(s) relative to the vehicle 10. The controller 26 is also configured to use the signal(s) 38A to determine a current position 40 of the vehicle 10 relative to the terrain 14. Generally, each GPS satellite continuously transmits a radio signal indicative of the current time and the satellite's position. Since the speed of radio waves is constant and independent of the GPS satellite speed, the time delay between when the satellite transmits a signal and the receiver receives it is proportional to the distance from the satellite to the receiver. The navigation module 30 typically monitors multiple satellites and solves equations to determine the precise position of the vehicle 10 and its deviation from true time. The navigation module 30 generally requires a minimum of four GPS satellites to be in view for the module to compute three position coordinates and the clock deviation from satellite time.
The controller 26 is additionally configured to determine a location 42 of a next waypoint 44, for example the external object 24, on the navigation route 34 and relative to the current position 40 of the vehicle 10. The controller 26 is also configured to issue a test query to the sensor 20 and receive therefrom an image frame 46 displaying a text 48 indicative of the next waypoint 42. The text 48 on the image frame 46 may, for example, be word(s) on a traffic, street, or business sign. The controller 26 is additionally configured to recognize and correlate the detected text 48 to the next waypoint 44 on the map 32 of the terrain 14. Furthermore, the controller 26 is configured to set an in-vehicle, such as inside the cabin 12A, alert 50 indicative of the detected text 48 having been correlated to the next waypoint 44.
The electronic controller 26 may be additionally configured to determine a distance 52 from the current position 40 of the vehicle 10 to the determined location 42 of the next waypoint 44. The electronic controller 26 may be additionally configured to determine whether the distance 52 from the current position 40 to the determined location 42 is within a threshold distance 54. The controller 26 may be further configured to set the alert 50 when the current position 40 of the vehicle 10 is within the threshold distance 54 of the next waypoint 44 and the text 48 is correlated to the next waypoint. The alert 50 may be set in a variety of audio and/or visual ways, each configured to indicate to the vehicle operator that the vehicle 10 is approaching the next waypoint 44.
With continued reference to
Generally, convolutional neural networks are used for image recognition. Convolutional neural networks employ fully-connected convolutional layers that work by learning a small set of weights which are applied one at a time as filters to small parts of the image. Weights are stored (in the convolutional layers) in a small matrix (often 3Ă—3) which is dot producted with each pixel, i.e., the scalar quantities are multiplied, to produce a new pixel, thus acting as image filters. The new images produced by each neuron/filter in a convolutional layer are then combined and passed as the inputs to every neuron in the next layer, and so on until the end of the neural network is reached. There is often a single dense layer at the end of a convolutional neural network to turn the image output of the final convolutional layer into the numerical class prediction that the neural network is being trained to produce. A fully-convolutional neural network is much like a convolutional neural network, but without fully-connected layers, i.e., it is comprised purely of convolutional layers and possibly some max-pooling layers. The output layer of a fully-convolutional neural network is a convolutional layer, and the output of such a neural network is therefore an image.
As shown, the first Neural Network 56 includes multiple layers 56-2, while the second Neural Network 58 includes multiple layers 58-2. Outputs from multiple layers 58-2 are merged with corresponding layers 56-2 in the first Neural Network 56 using at least one fully-connected layer 58-2A. Discrete values generated by layers 58-2 may be added element-wise to respective layers 56-2, i.e., individually, element by element. The first Neural Network 56 and second Neural Network 58 are trained together to output a mask score 60 having the recognized text 48 located on the recognized image frame 46. Of note, although the second Neural Network 58 is specifically disclosed herein as a 1-dimensional convolutional model, a bi-direction recurrent neural network, or another word representation model may also be used.
As shown in
The data gathering and processing system 18 may include a head-up display (HUD) 70 generally used to project select vehicle data inside the cabin 12A to inform the vehicle operator thereof. Specifically, the electronic controller 26 may set the in-vehicle alert 50 using HUD 70 to project a visual signal, such as a highlight icon 60A representative of the mask score 60, onto a view of the next waypoint 44. Such a visual signal may, for example, be projected onto a view of the next waypoint 44 in a vehicle windshield 72 or one of the side windows 74, in response to the detected orientation of the vehicle occupant's eyes, i.e., when the next waypoint 44 comes into the field of vision 62. In addition to HUD 70, the highlight icon 60A may be projected via a micro-electro-mechanical systems (MFMS) mirror 66 and light emitting diodes (LEDs) 68 embedded in the vehicle's dashboard 12B or A pillar 12C, and reflecting into the vehicle occupant's field of vision 62 by the vehicle's windshield 72.
To affect projection of the icon 60A onto the view of the next waypoint 44, and thereby highlight the corresponding external object 24, in a particular example, the vehicle 10 may include a fluorescent film 76 with laser excitation attached to vehicle windshield 72 (shown in
Following frame 106, the method may proceed to frame 108 or to frame 112. In frame 108 the method may include determining the distance 52 from the current position to the determined location 42 of the next waypoint 44, and then move to frame 110 for determining whether the distance 52 to the location 42 is within the threshold distance 54. If it is determined that the distance 52 of the determined location 42 of the next waypoint 44 is outside the threshold distance 54, the method may return to frame 106. If, on the other hand, it is determined that the distance 52 from the current position to the location 42 of the next waypoint 44 is within the threshold distance 54, the method may advance to frame 112. In frame 112 the method includes detecting and communicating to the electronic controller 26, via the sensor 20, the image frame 46 displaying the text 48 indicative of the next waypoint 44.
Following frame 112, the method moves on to frame 114. In frame 114, the method includes correlating, via the electronic controller 26, the detected text 48 to the next waypoint 44 on the map 32 of the terrain 14. According to the method, correlating the detected text 48 to the next waypoint 44 may include using the trained Neural Network architecture 55. As described above with respect to
In frame 116 the method includes setting, via the electronic controller 26, the in-vehicle alert 50 indicative of the detected text 48 having been correlated to the next waypoint 44. Accordingly, setting of the in-vehicle alert 50 may be performed when the distance 52 from the current position to the determined location 42 of the next waypoint 44 is within the threshold distance 54. Furthermore, setting the in-vehicle alert 50 may include projecting, via the HUD 70, the highlight icon 60A representative of the mask score 60 onto the view of the next waypoint 44. Additionally, in frame 116 the method may include determining the field of vision 62 of an occupant of the vehicle and setting the in-vehicle alert 50 in response to the determined field of vision.
As described above with respect to
The detailed description and the drawings or figures are supportive and descriptive of the disclosure, but the scope of the disclosure is defined solely by the claims. While some of the best modes and other embodiments for carrying out the claimed disclosure have been described in detail, various alternative designs and embodiments exist for practicing the disclosure defined in the appended claims. Furthermore, the embodiments shown in the drawings or the characteristics of various embodiments mentioned in the present description are not necessarily to be understood as embodiments independent of each other. Rather, it is possible that each of the characteristics described in one of the examples of an embodiment can be combined with one or a plurality of other desired characteristics from other embodiments, resulting in other embodiments not described in words or by reference to the drawings. Accordingly, such other embodiments fall within the framework of the scope of the appended claims.
