Lane marking localization

Information

  • Patent Grant
  • 11852498
  • Patent Number
    11,852,498
  • Date Filed
    Wednesday, May 5, 2021
    3 years ago
  • Date Issued
    Tuesday, December 26, 2023
    4 months ago
Abstract
Various embodiments of the present disclosure provide a system and method for lane marking localization that may be utilized by autonomous or semi-autonomous vehicles traveling within the lane. In the embodiment, the system comprises a locating device adapted to determine the vehicle's geographic location; a database; a region map; a response map; a camera; and a computer connected to the locating device, database, and camera, wherein the computer is adapted to: receive the region map, wherein the region map corresponds to a specified geographic location; generate the response map by receiving information form the camera, the information relating to the environment in which the vehicle is located; identifying lane markers observed by the camera; and plotting identified lane markers on the response map; compare the response map to the region map; and generate a predicted vehicle location based on the comparison of the response map and region map.
Description
BACKGROUND OF THE INVENTION
Field of the Invention

The present disclosure relates to unmanned vehicle guidance and, more specifically, a system for and method of sensing a roadway lane.


Description of Related Art

Global Positioning System (“GPS”) technology is widely used as a means for locating an automobile upon a roadway. As autonomous and semi-autonomous vehicles become more advanced, accurately knowing the vehicle's position in the roadway becomes critical. For example, self-driving cars by Volvo and Tesla have been easily confused by faded lane markers and other shabby road conditions. Further, current GPS technology is inaccurate. To achieve a fully autonomous self-driving vehicle requires the ability of a computer to determine the vehicle's lateral position within a roadway with great precision. Additionally, advanced driver-assistance systems (“ADAS”) benefit greatly from this ability. For example, lane keeping assistance (“LKA”) systems, lane departure warning (“LDW”) systems, and lane change assistance systems would be greatly benefited by accurately knowing the vehicle's lateral position within a lane. Other examples of ADAS systems include adaptive cruise control, adaptive light control, anti-lock braking systems, automatic parking, blind spot monitoring, collision avoidance systems, intersection detection, lane departure warning systems, parking sensors, turning assistance, and wrong-way driving warning.


A vehicle may utilize various levels of autonomous driving. For example, a first level of autonomous driving may assist a human driver during some driving tasks such as steering or engine acceleration/deceleration. A second level of autonomous driving may conduct some steering and acceleration/deceleration while the human driver monitors the driving environment and controls the remaining driving tasks. Such a system is referred to as a partially automated system. A third level of autonomous driving may conduct driving tasks and monitor the driving environment, but the human driver must be ready to retake control when the automated system requests. Such a system is generally referred to as a conditionally automated system. A fourth level of autonomous driving may drive the vehicle and monitor road conditions; the human driver does not need to take control but the system may only operate in specific conditions and environments such as inside of a factory, on a closed road course, or within a bounded area. Such a system is referred to as a highly automated system. A fifth level of autonomous driving may perform all driving and road-monitoring tasks in all driving conditions. Such a system is referred to as a fully-automated system.


Current technology relies on GPS technology to determine a vehicle's lateral position within a roadway. However, this method is susceptible to a high amount of drift—the lateral area around the vehicle that is within the technology's margin of error. The amount of drift in a given system is dependent on many factors including signal strength and the precision of the GPS hardware being used. Typical GPS devices aimed at the average consumer have a drift of about IO meters. Even with the most precise instruments having the best signal strength, a system experiences a drift of I-2 meters or more, which is unacceptable for self-driving vehicles.


To improve the accuracy of GPS positioning, current technology also employs an inertial measurement unit (“IMU”). An IMU is an electronic device that measures and reports a vehicle's specific force and angular rate using a combination of accelerometers and gyroscopes. However, even while being augmented with IMU's, current lateral locating methods and systems still experience a high amount of drift. For such a system to be useful in a self-driving vehicle, the resolution needs to be approximately IO cm or less.


Therefore, what is needed is a system that can utilize GPS information and determine a vehicle's lateral position within a roadway with great accuracy. This need has heretofore remained unsatisfied.


SUMMARY OF THE INVENTION

The present disclosure overcomes these and other deficiencies of the prior art by providing a method for determining a vehicle's location comprising the steps of approximating the vehicle's region, receiving a region map from a database, wherein the region map corresponds to the vehicle's approximated region and comprises a plurality of region points indicating an expected roadway lane, receiving a response image generated by an imaging device, the response image comprising information relating to the vehicle's environment, generating a response map from the response image, the response map comprising a plurality of response points indicating the vehicle's location, comparing the response map to the region map, and predicting the vehicle's location based on the differences between the response points and the region points.


In another exemplary embodiment of the present disclosure, the vehicle's region may be approximated using a GPS device or an IMU device.


In another exemplary embodiment of the present disclosure, the step of generating a response map further comprises the steps of detecting lane markers in the response image, the lane markers pertaining to physical aspects contained in the response image, and plotting the response points on the response map, the response points indicating locations of the lane markers.


In another exemplary embodiment of the present disclosure, the method may further comprise the step of generating a confidence score.


In another exemplary embodiment of the present disclosure, the response image may be generated from radar sensing equipment, LIDAR sensing equipment, GPS sensing information, and/or images.


In another exemplary embodiment of the present disclosure, the region map and the response map may be compared at a selected frequency.


In another exemplary embodiment of the present disclosure, the selected frequency may be at least 20 cycles per second.


In another exemplary embodiment of the present disclosure, the method may further comprise the step of outputting the vehicle's predicted location to an ADAS.


In another exemplary embodiment of the preset disclosure, the imaging devices may comprise a plurality of imaging devices, each adapted to perceive different aspects of the vehicle's environment.


In overcoming the limitations currently available in the art, another exemplary embodiment provides a system for determining a vehicle's location on a roadway comprising a locating device adapted to determine a vehicle's geographic region, a database comprising a plurality of region maps, the region maps comprising a plurality of region points, an imaging device adapted to perceive information relating to the vehicle's environment, a processor operably connected to the locating device, the database, and the imaging device, the processor, at a predetermined frequency, adapted to receive, from the locating device, the vehicle's determined geographic region, receive, from the database, the region map corresponding to the vehicle's determined geographic region, receive, from the imaging device, information perceived relating to the vehicle's environment, generate a response map, the response map comprising a plurality of response points corresponding to lane markers detected within the response map, compare the response map to the region map, and determine the vehicle's predicted location based on the comparison of the region map and response map.


In another exemplary embodiment of the present disclosure, the locating device may comprise a GPS device or an IMU device.


In another exemplary embodiment of the present disclosure, the imaging device may comprise a camera or a LIDAR device.


In another exemplary embodiment of the present disclosure, the predetermined frequency is at least 20 cycles per second.


In another exemplary embodiment of the present invention, the processor may be further configured to output the vehicle's predicted location to an ADAS.


In another exemplary embodiment of the present disclosure, the processor may be further configured to determine a confidence score.


In another exemplary embodiment of the present disclosure, the system may further comprise a plurality of imaging devices, each adapted to perceive different aspects of the vehicle's environment.


In some embodiments, the present disclosure provides various vehicle systems the ability to recognize and track lane markings, which, in turn, may be used to allow on-board monitors to detect and/or correct the vehicle's location. For example, the present disclosure may interact with a lane keeping assistance (“LKA”) system, lane departure warning (“LDW”) system, and lane change assistance system. Such examples may utilize the present disclosure because such systems need to know the relative position the vehicles within the ego-lane and/or the lane to the left/right.


The foregoing, and other features and advantages of the invention, will be apparent from the following, more particular description of the preferred embodiments of the invention, the accompanying drawings, and the claims.





BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure, the objects and advantages thereof, reference is now made to the ensuing descriptions taken in connection with the accompanying drawings briefly described as follows:



FIG. 1 illustrates a system for determining a vehicle's position within a lane, according to an exemplary embodiment of the present disclosure;



FIG. 2 illustrates a method for determining a vehicle's position within a lane, according to an exemplary embodiment of the present disclosure;



FIG. 3A illustrates an image taken by the camera, according to an exemplary embodiment of the present disclosure;



FIG. 3B illustrates a response map, according to an exemplary embodiment of the present disclosure;



FIG. 3C illustrates a lane map, according to an exemplary embodiment of the present disclosure; and



FIG. 4 illustrates updated vehicle location, according to an exemplary embodiment of the present disclosure.





DETAILED DESCRIPTION OF EMBODIMENTS

Further features and advantages of the disclosure, as well as the structure and operation of various embodiments of the disclosure, are described in detail below with reference to the accompanying FIGS. 1-4. Although the disclosure is described in the context of a generic vehicle, the term vehicle refers to any type of motorized ground transport including, but not limited to, cars, trucks, carts, sleds, lifts, and rovers.


In an exemplary embodiment of the present disclosure, the system utilizes hardware including a camera, a database, and a computer, to iteratively update the vehicle's predicted location and to determine the vehicle's position relative to the lane of traffic in which it is traveling. In one embodiment, the camera is installed on the vehicle and its position and view angle are predetermined relative to the rest of the vehicle on which it is installed. For example, the camera may be installed on the roof of the vehicle at the centerline of the vehicle, and pointed in the direction of travel, i.e., forward, such that the camera is out of the view of the driver. The computer is configured to include the camera's position and orientation relative to the vehicle. The computer fetches data from the camera and generates a response map. The response map is generated by identifying and locating lane indicators depicted the camera's data. The computer fetches data from the database, including a region map. The region map comprises information previously gathered by a collection vehicle equipped with lane sensing devices including radar, LIDAR, GPS, and cameras. In such an embodiment, the collection vehicle, along with the equipment thereon, accurately determines the collection vehicle's location in relation to lane markers. Such lane markers include traditional lane markings such as lines painted in a roadway and reflectors. Lane markers may also include permanent or semi-permanent structures such as raised curbs, barricades, retaining walls, roadway shoulders, roadway barriers, bridges, buildings, street signs, tunnels, trees, any support structures thereof.


The present disclosure utilizes the collection vehicle's accurately determined roadway position relative to lane markers as a baseline to determine a subsequent vehicle's location within the same roadway at a similar longitudinal roadway position. For example, the subsequent vehicle may be referred to as the “target vehicle.” As the target vehicle moves down a roadway, it captures information to generate the response map. The response map comprises information similar to that of a region map. The target vehicle may have a lateral position within the roadway different from that of the collection vehicle's lateral position at the same longitudinal roadway position. The present disclosure determines the target vehicle's location within the roadway by comparing the response map—the location information captured by the target vehicle—against the region map—the location information captured by collection vehicle. In doing so, the target vehicle's lateral roadway position is accurately determined relative to (1) the lane indicators perceived by target vehicle and (2) the collection vehicle's accurately determined roadway position relative to the same lane indicators.


The computer compares the response map against the region map to determine a confidence score. The confidence score is a numerical representation of the differences between the response map and the region map; the greater the difference, the lower the confidence score. If the confidence score is below a predetermined threshold, the computer generates the vehicle's predicted position. The predicted position is generated by comparing the region map against the response map and determining the vehicle's likely position based on the relative perspectives of the response map and the region map. In another embodiment, the computer generates the vehicle's predicted position at any confidence score. The system may output the updated location information to another system. For example, the system may output the updated location information to the vehicle's self-driving automation system and/or an ADAS system installed on the vehicle. In another example, the system may output the updated location information to a database or another vehicle. Such a system may be implemented as part of an automated self-driving system that steers the vehicle in response to the updated location information. In another embodiment, such a system may be implemented as part of an ADAS.


In an exemplary embodiment of the present disclosure and with reference to FIG. 1, a system 100 is utilized as part of a vehicle to determine the vehicle's location within a lane. The system may comprise a GPS device 120 to provide an approximation of the vehicle's actual location and an IMU 110 to estimate the vehicle's speed and velocity. The system may also comprise a database 140 comprising environment data 130. The environment data 130 may comprise a plurality of region maps. The system may also comprise a camera 150 configured to perceive information relating to the vehicle's environment. The system may also comprise a computer 160 and an output device 170.


In an embodiment, the GPS device 120 gathers information comprising the latitude, longitude, and/or the position of the vehicle. In another embodiment, the GPS information comprises the vehicle's yaw angle and/or direction of travel. In another embodiment, the GPS information comprises a time and date stamp. The GPS device 120 may receive information from orbiting satellites that are widely used in the current state of the art. General GPS signaling and positioning is readily apparent to one skilled in the art.


The database 140 comprises environment data 130 obtained using GPS and/or radar, and/or light detection and ranging (“LIDAR”), and/or photographic cameras, and/or video graphic cameras. In such an embodiment, a collection vehicle (not shown) comprises equipment including radar, LIDAR, GPS, and cameras. The equipment installed on the collection vehicle accurately gathers location information as it is driven down a roadway. This location information is uploaded to the database 140 and is used to create the environment data 130. The environment data 130 is used to identify physical structures that aid the system in determining a vehicle's position relative to those structures. For example, the environment data 130 may comprise location information relating to lane markers including lane separators, lane markings, and reflectors. In such an embodiment, the collection vehicle accurately determines its physical location relative to the lane markers. In another embodiment, the environment data 130 further comprises data relating to permanent or semi-permanent structures including bridges, signs, buildings, barriers, street lights, raised curbs, trees, support structures thereof, and other physical structures. In one embodiment, the database 140 is located remotely. In another embodiment, the database 140 is located on the vehicle. In another embodiment, the database 140 may comprise a plurality of local or remote databases, communicatively connected to one-another and to the computer 160.


The camera 150 may be installed on the vehicle having any orientation or view angle. For example, the camera 150 may be installed such that it points in the direction of travel, i.e., towards the front of the vehicle. In another embodiment, the camera 150 may be installed such that it points in a direction other than the direction of travel, i.e., towards the rear or sides of the vehicle. In one embodiment, the camera 150 comprises a video camera gathering video at a predetermined rate. In another embodiment, the camera 150 comprises a video camera with frame rate of at least 20 frames per second. In another embodiment, the camera 150 comprises a photographic camera capturing images at a predetermined rate. In an embodiment, the camera 150 comprises a photographic camera capturing images at rate of at least 10 frames per second. In another embodiment, the camera 150 may be configured to capture information beyond that visible to the human eye. For example, the camera 150 may be configured to capture infrared light and/or ultraviolet light. While a camera is disclosed herein, the disclosed subject matter is not limited to a camera comprising a lens and/or a light sensor or film. For example, the camera 150 may be a depth sensor. In such an embodiment, the camera 150 comprises a light-emitting device and a sensor capable of detecting the light emitted from that device. In such an embodiment, light-emitting device emits a plurality of beams of light, for example, infrared laser beams. The plurality of infrared lasers reflect light off of various surfaces and structures, for example, roadway reflectors and lane markers. The camera's 150 sensor detects the infrared lasers and generates a depth map of the environment perceived by the camera 150. Further, as disclosed here, the camera 150 may comprise a plurality of cameras pointed in the same or differing directions. In such an embodiment, the plurality of cameras may be installed at different locations on the vehicle.


The present disclosure includes a computer 160 for processing the data from the GPS 120, the database 140, and the camera 150. The computer 160 generates the vehicle's predicted location by comparing the vehicle's approximate location fetched from the GPS 120, the environment data 130 fetched from the database 140, and the information gathered from the camera 150. The environment data may include a region map (not shown). In one embodiment, the system uses the GPS 120 to determine an approximate location of the vehicle. The computer 160 fetches environment data 130 relating to the vehicle's approximate location, as determined by the GPS 120. The computer 160 fetches data from the camera 150. The computer 160 determines the vehicle's predicted location by comparing the environment data 130 against the data fetched from the camera 150.


In another embodiment, the computer 160 also determines a confidence score that correlates to how well the data fetched from the camera 150 matches the environment data 130 fetched from the GPS 120 and the database 140.


After the computer 160 determines the vehicle's predicted location, the computer 160 may output that information to an output device 170. For example, the computer 160 may output the vehicle's predicted location to self-driving automation system. In another embodiment, the computer 160 may output the vehicle's predicted location to an ADAS. In another embodiment, the computer 160 may output the vehicle's predicted location to a database.



FIG. 2 illustrates a method 200 for determining a vehicle's position within a lane, according to an exemplary embodiment of the disclosure. At step 210, a computer fetches GPS and/or IMU location information from an GPS device and/or an IMU device. At step 220, the computer generates a region map comprising previously-gathered information relating to the environment in which the vehicle is traveling. For example, the region map comprises information previously gathered by a collection vehicle using radar, LIDAR, GPS, and/or cameras. Such information pertained to the collection vehicle's location on a specific roadway relative to other roadways in the area, lane-specific information relative to the lane in which the collection vehicle is traveling, and information relating to the collection vehicle's speed, direction of travel, and/or velocity relative to the location information. In one embodiment, the computer generates the region map. In another embodiment, the computer receives the region map from a database.


At step 230, the system utilizes a camera installed on the vehicle. In one embodiment, the camera is installed on the vehicle having a predetermined viewing angle and orientation. For example, the camera is installed on the roof of the vehicle, centered on the vehicle's centerline, and pointing in the direction of travel, i.e., forward. The camera captures an image of the region in front of the vehicle. In another embodiment, the camera may capture video and/or photographic images at a predetermined frame rate. In another embodiment, the camera captures infrared and/or ultraviolet light. In one embodiment, the camera captures images at a predetermined rate. In another example, the camera captures images at a rate of at least 10 images per second.


At step 240, the system generates a response map based on information fetched from the camera. The response map may be generated in real-time or in near real-time. The response map may be generated on a predetermined interval, for example, 20 times per second. In one embodiment, the system uses an image fetched from the camera and identifies lane markers within the lanes of vehicle travel depicted in the image. The camera may identify other aspects of the roadway including, but not limited to, bridges, signs, barriers, street lights, and buildings. In one embodiment, the computer comprises computer-executable code configured to detect permanent and/or semi-permanent structures within a two-dimensional image. In such an embodiment, the computer analyzes the image captured from the camera and identifies lane indicators such as painted lines and reflectors. The computer may also identify other structures such as bridges, signs, barriers, street lights, and buildings. The computer may generate a response map on a predetermined interval. In one embodiment, the computer generates a response map at least ten times per second.


At step 250, the system generates the vehicle's predicted location and calculates a confidence score for determining the vehicle's lateral position within a lane. For example, the system determines the predicted location by comparing the region map against the response map. In such an embodiment, the system samples various points within the region map identifying lanes of vehicle travel. The system samples the response map and identifies lanes of travel depicted therein. The system then compares this sampled region map to the response map and generates the vehicle's predicted location based on the differences in the perspectives of the region and response maps. In such an embodiment, the system takes the GPS/IMU information, the region map, and the response map as arguments in calculating the vehicle's predicted location. For example, if the region map is substantially the same as the response map but skewed to the left, the system's comparison recognizes the vehicle's actual position must be to the right of the GPS location. The system generates a predicted vehicle location based those differences.


In another embodiment, at step 250, the system calculates a confidence score. In one embodiment, for example, where the region map and the response map are identical, the system generates a confidence score of 1.000. In such an example, the environment data was gathered using a collection vehicle that was located at the same physical location with the same orientation of that of the system's vehicle. The confidence score reflects the system's confidence in the vehicle's predicted position compared to its position according to the region map, relative to the vehicle's lateral position within a lane. For example, a score of 1.000 correlates to a confidence of 100% and a score of 0.000 correlates to a confidence of 0%. At step 260, the system outputs a predicted location. In one embodiment, the system may output the predicted location to an automated self-driving system. In another embodiment, the system may output the predicted location to an ADAS. In another embodiment, the system may output a corrected location if the confidence score is below a predetermined threshold. For example, the score threshold is set at 0.900. If the system generates a confidence score of anything less than 0.900, for example, a score of 0.85, the system generates a corrected location based on the comparison of the sampled region map and the response map. In an embodiment, the mathematical variance may be used as a confidence score. Further, if the system generates a confidence score of, for example, 0.950, the system outputs the vehicle's position as determined by the GPS/IMU information. In another embodiment, the system outputs the corrected location to an ADAS and/or an automated self-driving system. In another embodiment, the mathematical variance is used as the confidence score.



FIG. 3A illustrates an image taken by a single camera 120. Here, the camera 120 is facing the direction of travel. In other embodiments of the disclosure, the camera is positioned such that the view captured is not the direction of travel, e.g., facing behind the vehicle or to either side. In other embodiments of the disclosure, a plurality of cameras may be used. As disclosed herein, the camera may be installed anywhere on the vehicle having any orientation that allows the camera to view the vehicle's environment. When the camera is installed on the vehicle, the system may be updated as to the camera's positioning relative to the rest of the vehicle and the direction of travel. The system 100 analyzes the image taken by the camera and creates a response map by detecting lane markers such as sold lines 301, striped lines 303, and reflectors 302. In one embodiment, the camera may be permanently installed on the vehicle. For example, the camera may be integrated into the vehicle's rearview mirror or a bumper. In another embodiment, the camera may be temporarily installed on the vehicle. In another embodiment, the camera utilized may be included in a mobile device such as a cell phone or tablet. In such an embodiment, the mobile device may be temporarily installed on the vehicle and easily removed by a user.



FIG. 3B illustrates an exemplary response map according to an embodiment of the disclosure. The response map reflects lane markings as recognized by the system. For example, the response map is a binary map indicating lane markings 311 shown as black lines. In such an embodiment, the system analyzes the location information for lane indicators. When the system identifies a lane indicator, it plots a point on the response map (depicted as a black dot against a white background). Everything other than the relevant lane markings 311 are shown as white space 312. The system plots a plurality of lane indicators on the response map, culminating as lane markers 311. The system may also use physical structures such as bridges, barriers, signs, and buildings to determine lane markings.


In one embodiment, the computer 160 comprises computer-executable, non-transient code configured to detect certain elements with an image. For example, the computer 160 recognizes lane markings within a roadway including painted solid lines 301, painted striped lines 303, and reflectors 302. The system generates the response map as a series points, culminating a lane marking lines 311. The response map represents the road ahead of the vehicle, viewed from the camera 150 and perceived by the computer 160. In other embodiments, the lane markings 311 reflect other structural components such as bridges, signs, and barriers (not shown).



FIG. 3C illustrates a lane map, according to an embodiment of the present disclosure. The lane map comprises a comparison of the region map against the response map. The region map is compiled using images, GPS, radar, and/or LIDAR information. The system plots plurality of region map points 322 (shown as circles). The region map points 322 reflect lane markings as detected in the environment data 130. The response map comprises a plurality lines 321 (shown as grey lines) indicating the lane markers as viewed from the camera and perceived by the computer. The computer analyzes the differences between the region map and the response map and generates the vehicle's predicted location.


In another embodiment, the system determines a confidence score based on the differences in the lane map, which is shown in the top-left corner. For example, a perfect match overlays with 100% accuracy, resulting in a score of 1.000 (not shown). In another example, the system may determine a score of 0.74 where the overlay is a 74% match (as shown). In such an embodiment, the overlay is close, but the region map points 322 differs from the points from the response map lines 321 at some, but not all of the region map points 322. In such an embodiment, the score threshold may be 0.90, and in such an instance, the system would output a predicted vehicle location by analyzing the differences in the lane map. In another embodiment, the system may also determine other statistical parameters, such as the variance. In such an embodiment, the variance is calculated, for example, of 0.384 (as shown). For example, a logistic function may be used to calculate the variance, such as:






variance
=



x
min

+

(


x
max

-

x
min


)



1
+

e

S

(

Gx
-
m

)








where,


xmin=the minimum value


xmax=, the maximum value


S=the steepness


G=the growth rate


x=the matching score of the response map


m=the midpoint



FIG. 4 illustrates an updated vehicle location, according to an embodiment of the present disclosure. The predicted location 401 is determined using GPS and/or IMU locating information. The GPS location 402 represents where the GPS perceives the vehicle to be relative to the lanes of travel. Notably, the GPS location 402 often varies greatly from the vehicle's actual location 403. Where the score is below a predetermined threshold, the system determines the predicted location 401 and outputs that location. In such an embodiment, the system recognizes which lane of traffic the vehicle is traveling. In other embodiments, the system determines the vehicle's predicted location 401 regardless of the confidence score. In other embodiments, the system may use the confidence score to determine the vehicle's predicted location 401.


Although a lane marking localization system has been shown and described, lane marking localization systems may be implemented according to other embodiments of the disclosure. For example, the system may utilize a plurality of cameras or other information gathering devices such as radar or LIDAR. Other embodiments of the disclosure may utilize a plurality of external or internal databases, on which relevant information is stored. Other embodiments also include those that output information to vehicle driving aids such as navigation and ADAS systems.


In an embodiment of the disclosure, the methodologies and techniques described herein are implemented on a special purpose computer programmed to determine lane marking and relative vehicle position. In an embodiment of the disclosure, the special-purpose computer comprises an embedded system with a dedicated processor equipped as part of a vehicle. In other embodiments, some or all of the components of the present disclosure may be integrated as part of a mobile device, for example, a cell phone or a tablet. The disclosure has been described herein using specific embodiments for the purposes of illustration only. It will be readily apparent to one of ordinary skill in the art, however, that the principles of the disclosure can be embodied in other ways. Therefore, the disclosure should not be regarded as being limited in scope to the specific embodiments disclosed herein, but instead as being fully commensurate in scope with the following claims.

Claims
  • 1. A method implemented by a processor, comprising: approximating a vehicle's region;receiving a region map from a database, wherein the region map corresponds to the vehicle's approximated region and comprises a plurality of region points indicating an expected roadway lane;receiving a response image generated by an imaging device of one or more imaging devices of the vehicle, the response image comprising information relating to the vehicle's environment;generating a response map from the response image, the response map comprising a plurality of response points indicating the vehicle's location;comparing the response map to the region map to determine (a) differences between the plurality of response points and the plurality of region points, and (b) a confidence score that comprises a variance that is computed using a logistic function; andpredicting the vehicle's roadway position based on the confidence score and the differences between the plurality of response points and the plurality of region points.
  • 2. The method of claim 1, wherein the vehicle's region is approximated using a Global Positioning System (GPS) device or an inertial measurement unit (IMU) device.
  • 3. The method of claim 1, wherein the generating the response map further comprises: detecting lane markers in the response image, the lane markers pertaining to physical aspects contained in the response image; andplotting the response points on the response map, the response points indicating locations of the lane markers.
  • 4. The method of claim 1, further comprising: generating, based on the differences, the confidence score.
  • 5. The method of claim 1, wherein the response image is generated from radar sensing equipment, light detection and ranging (LIDAR) sensing equipment, Global Positioning System (GPS) sensing information, and/or images.
  • 6. The method of claim 1, wherein the region map and the response map are compared at a selected frequency.
  • 7. The method of claim 6, wherein the selected frequency is at least 20 cycles per second.
  • 8. An apparatus, comprising: a processor configured to: approximate a vehicle's region;receive a region map from a database, wherein the region map corresponds to the vehicle's approximated region and comprises a plurality of region points indicating an expected roadway lane;receive a response image generated by an imaging device of one or more imaging devices of the vehicle, the response image comprising information relating to the vehicle's environment;generate a response map from the response image, the response map comprising a plurality of response points indicating the vehicle's location;compare the response map to the region map to determine (a) differences between the plurality of response points and the plurality of region points, and (b) a confidence score that comprises a variance that is computed using a logistic function; andpredict the vehicle's roadway position based on the confidence score and the differences between the plurality of response points and the plurality of region points.
  • 9. The apparatus of claim 8, wherein the processor is further configured to: output the vehicle's predicted location to an advanced driver-assistance system (ADAS).
  • 10. The apparatus of claim 8, wherein each of the one or more imaging devices is adapted to perceive a different aspect of the vehicle's environment.
  • 11. The apparatus of claim 8, wherein the variance is computed as:
  • 12. The apparatus of claim 8, wherein the imaging device comprises a light detection and ranging (LIDAR) device.
  • 13. The apparatus of claim 8, wherein the imaging device comprises a camera that captures images at a rate of at least 10 images per second.
  • 14. A non-transitory computer-readable storage medium having instructions stored thereupon, the storage medium comprising: instructions for approximating a vehicle's region;instructions for receiving a region map from a database, wherein the region map corresponds to the vehicle's approximated region and comprises a plurality of region points indicating an expected roadway lane;instructions for receiving a response image generated by an imaging device of one or more imaging devices of the vehicle, the response image comprising information relating to the vehicle's environment;instructions for generating a response map from the response image, the response map comprising a plurality of response points indicating the vehicle's location;instructions for comparing the response map to the region map to determine (a) differences between the plurality of response points and the plurality of region points, and (b) a confidence score that comprises a variance that is computed using a logistic function; andinstructions for predicting the vehicle's roadway position based on the confidence score and the differences between the plurality of response points and the plurality of region points.
  • 15. The storage medium of claim 14, further comprising: instructions for comparing the confidence score to a threshold; andinstructions for outputting the vehicle's predicted roadway position based on the confidence score being lower than the threshold.
  • 16. The storage medium of claim 14, wherein the response map is generated using a camera pointing in a direction opposite to a direction of travel of the vehicle.
  • 17. The storage medium of claim 14, wherein lane markers in the response map are based on one or more physical structures.
  • 18. The storage medium of claim 17, wherein the one or more physical structures comprise a bridge, a barrier, a sign, or a building.
  • 19. The storage medium of claim 17, wherein the lane markers comprise at least one of solid lines, striped lines, or reflectors.
  • 20. The storage medium of claim 14, wherein the region map further comprises information related to a speed of the vehicle, a direction of travel of the vehicle, or a velocity of the vehicle relative to the expected roadway lane.
CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No. 15/896,077, entitled “LANE MARKING LOCALIZATION”, filed Feb. 14, 2018. The entire content of the above-mentioned patent application is incorporated by reference as part of the disclosure of this document.

US Referenced Citations (260)
Number Name Date Kind
6084870 Wooten et al. Jul 2000 A
6263088 Crabtree et al. Jul 2001 B1
6594821 Banning et al. Jul 2003 B1
6777904 Degner et al. Aug 2004 B1
6975923 Spriggs Dec 2005 B2
7103460 Breed Sep 2006 B1
7689559 Canright et al. Mar 2010 B2
7742841 Sakai et al. Jun 2010 B2
7783403 Breed Aug 2010 B2
7844595 Canright et al. Nov 2010 B2
8041111 Wilensky Oct 2011 B1
8064643 Stein et al. Nov 2011 B2
8082101 Stein et al. Dec 2011 B2
8164628 Stein et al. Apr 2012 B2
8175376 Marchesotti et al. May 2012 B2
8271871 Marchesotti Sep 2012 B2
8346480 Trepagnier et al. Jan 2013 B2
8378851 Stein et al. Feb 2013 B2
8392117 Dolgov et al. Mar 2013 B2
8401292 Park et al. Mar 2013 B2
8412449 Trepagnier et al. Apr 2013 B2
8478072 Aisaka et al. Jul 2013 B2
8553088 Stein et al. Oct 2013 B2
8706394 Trepagnier et al. Apr 2014 B2
8718861 Montemerlo et al. May 2014 B1
8788134 Litkouhi et al. Jul 2014 B1
8908041 Stein et al. Dec 2014 B2
8917169 Schofield et al. Dec 2014 B2
8963913 Baek Feb 2015 B2
8965621 Urmson et al. Feb 2015 B1
8981966 Stein et al. Mar 2015 B2
8983708 Choe et al. Mar 2015 B2
8993951 Schofield et al. Mar 2015 B2
9002632 Emigh Apr 2015 B1
9008369 Schofield et al. Apr 2015 B2
9025880 Perazzi et al. May 2015 B2
9042648 Wang et al. May 2015 B2
9081385 Ferguson et al. Jul 2015 B1
9088744 Grauer et al. Jul 2015 B2
9111444 Kaganovich Aug 2015 B2
9117133 Barnes et al. Aug 2015 B2
9118816 Stein et al. Aug 2015 B2
9120485 Dolgov Sep 2015 B1
9122954 Srebnik et al. Sep 2015 B2
9134402 Sebastian et al. Sep 2015 B2
9145116 Clarke et al. Sep 2015 B2
9147255 Zhang et al. Sep 2015 B1
9156473 Clarke et al. Oct 2015 B2
9176006 Stein Nov 2015 B2
9179072 Stein et al. Nov 2015 B2
9183447 Gdalyahu et al. Nov 2015 B1
9185360 Stein et al. Nov 2015 B2
9191634 Schofield et al. Nov 2015 B2
9214084 Grauer et al. Dec 2015 B2
9219873 Grauer et al. Dec 2015 B2
9233659 Rosenbaum et al. Jan 2016 B2
9233688 Clarke et al. Jan 2016 B2
9248832 Huberman Feb 2016 B2
9248835 Tanzmeister Feb 2016 B2
9251708 Rosenbaum et al. Feb 2016 B2
9277132 Berberian Mar 2016 B2
9280711 Stein Mar 2016 B2
9282144 Tebay et al. Mar 2016 B2
9286522 Stein et al. Mar 2016 B2
9297641 Stein Mar 2016 B2
9299004 Lin et al. Mar 2016 B2
9315192 Zhu et al. Apr 2016 B1
9317033 Ibanez-Guzman et al. Apr 2016 B2
9317776 Honda et al. Apr 2016 B1
9330334 Lin et al. May 2016 B2
9342074 Dolgov et al. May 2016 B2
9347779 Lynch May 2016 B1
9355635 Gao et al. May 2016 B2
9365214 Ben Shalom et al. Jun 2016 B2
9399397 Mizutani et al. Jul 2016 B2
9418549 Kang et al. Aug 2016 B2
9428192 Schofield et al. Aug 2016 B2
9436880 Bos et al. Sep 2016 B2
9438878 Niebla et al. Sep 2016 B2
9443163 Springer Sep 2016 B2
9446765 Ben Shalom et al. Sep 2016 B2
9459515 Stein Oct 2016 B2
9466006 Duan Oct 2016 B2
9476970 Fairfield Oct 2016 B1
9483839 Kwon et al. Nov 2016 B1
9490064 Hirosawa et al. Nov 2016 B2
9494935 Okumura et al. Nov 2016 B2
9507346 Levinson et al. Nov 2016 B1
9513634 Pack et al. Dec 2016 B2
9531966 Stein et al. Dec 2016 B2
9535423 Debreczeni Jan 2017 B1
9538113 Grauer et al. Jan 2017 B2
9547985 Tuukkanen Jan 2017 B2
9549158 Grauer et al. Jan 2017 B2
9552657 Ueno Jan 2017 B2
9555803 Pawlicki et al. Jan 2017 B2
9568915 Berntorp et al. Feb 2017 B1
9587952 Slusar Mar 2017 B1
9599712 Van Der Tempel et al. Mar 2017 B2
9600889 Boisson et al. Mar 2017 B2
9602807 Crane et al. Mar 2017 B2
9612123 Levinson et al. Apr 2017 B1
9620010 Grauer et al. Apr 2017 B2
9625569 Lange Apr 2017 B2
9628565 Stenneth et al. Apr 2017 B2
9649999 Amireddy et al. May 2017 B1
9652860 Maali et al. May 2017 B1
9669827 Ferguson et al. Jun 2017 B1
9672446 Vallespi-Gonzalez Jun 2017 B1
9690290 Prokhorov Jun 2017 B2
9701023 Zhang et al. Jul 2017 B2
9712754 Grauer et al. Jul 2017 B2
9720418 Stenneth Aug 2017 B2
9723097 Harris et al. Aug 2017 B2
9723099 Chen et al. Aug 2017 B2
9723233 Grauer et al. Aug 2017 B2
9726754 Massanell et al. Aug 2017 B2
9729860 Cohen et al. Aug 2017 B2
9738280 Rayes Aug 2017 B2
9739609 Lewis Aug 2017 B1
9746550 Nath et al. Aug 2017 B2
9753128 Schweizer et al. Sep 2017 B2
9753141 Grauer et al. Sep 2017 B2
9754490 Kentley et al. Sep 2017 B2
9760837 Nowozin et al. Sep 2017 B1
9766625 Boroditsky et al. Sep 2017 B2
9769456 You et al. Sep 2017 B2
9773155 Shotton et al. Sep 2017 B2
9779276 Todeschini et al. Oct 2017 B2
9785149 Wang et al. Oct 2017 B2
9805294 Liu et al. Oct 2017 B2
9810785 Grauer et al. Nov 2017 B2
9823339 Cohen Nov 2017 B2
9842399 Yamaguchi Dec 2017 B2
9953236 Huang et al. Apr 2018 B1
10147193 Huang et al. Dec 2018 B2
10223806 Luo et al. Mar 2019 B1
10223807 Luo et al. Mar 2019 B1
10410055 Wang Sep 2019 B2
10529089 Ahmad et al. Jan 2020 B2
10698100 Becker Jun 2020 B2
10816354 Liu Oct 2020 B2
20010051845 Itoh Dec 2001 A1
20030114980 Klausner et al. Jun 2003 A1
20030174773 Comaniciu et al. Sep 2003 A1
20040264763 Mas et al. Dec 2004 A1
20070088497 Jung Apr 2007 A1
20070183661 El-Maleh et al. Aug 2007 A1
20070183662 Wang et al. Aug 2007 A1
20070230792 Shashua et al. Oct 2007 A1
20070286526 Abousleman et al. Dec 2007 A1
20080109118 Schwartz May 2008 A1
20080249667 Horvitz et al. Oct 2008 A1
20090040054 Wang et al. Feb 2009 A1
20090087029 Coleman et al. Apr 2009 A1
20090243825 Schofield Oct 2009 A1
20100049397 Liu et al. Feb 2010 A1
20100082238 Nakamura Apr 2010 A1
20100111417 Ward et al. May 2010 A1
20100226564 Marchesotti et al. Sep 2010 A1
20100281361 Marchesotti Nov 2010 A1
20110142283 Huang et al. Jun 2011 A1
20110206282 Aisaka et al. Aug 2011 A1
20110247031 Jacoby Oct 2011 A1
20120041636 Johnson et al. Feb 2012 A1
20120105639 Stein et al. May 2012 A1
20120120069 Kodaira et al. May 2012 A1
20120140076 Rosenbaum et al. Jun 2012 A1
20120274629 Baek Nov 2012 A1
20120314070 Zhang et al. Dec 2012 A1
20130051613 Bobbitt et al. Feb 2013 A1
20130083959 Owechko et al. Apr 2013 A1
20130182134 Grundmann et al. Jul 2013 A1
20130204465 Phillips et al. Aug 2013 A1
20130266187 Bulan et al. Oct 2013 A1
20130329052 Chew Dec 2013 A1
20140063489 Steffey Mar 2014 A1
20140072170 Zhang et al. Mar 2014 A1
20140104051 Breed Apr 2014 A1
20140142799 Ferguson et al. May 2014 A1
20140143839 Ricci May 2014 A1
20140145516 Hirosawa et al. May 2014 A1
20140198184 Stein et al. Jul 2014 A1
20140314322 Snavely Oct 2014 A1
20140321704 Partis Oct 2014 A1
20140334668 Saund Nov 2014 A1
20150062304 Stein et al. Mar 2015 A1
20150127239 Breed May 2015 A1
20150253428 Holz Sep 2015 A1
20150269437 Maruyama et al. Sep 2015 A1
20150269438 Samarasekera et al. Sep 2015 A1
20150292891 Kojo Oct 2015 A1
20150310370 Burry et al. Oct 2015 A1
20150353082 Lee et al. Dec 2015 A1
20160008988 Kennedy et al. Jan 2016 A1
20160026787 Nairn et al. Jan 2016 A1
20160037064 Stein et al. Feb 2016 A1
20160046290 Aharony et al. Feb 2016 A1
20160094774 Li et al. Mar 2016 A1
20160118080 Chen Apr 2016 A1
20160125608 Sorstedt May 2016 A1
20160129907 Kim et al. May 2016 A1
20160165157 Stein et al. Jun 2016 A1
20160191860 Jung Jun 2016 A1
20160210528 Duan Jul 2016 A1
20160275766 Venetianer et al. Sep 2016 A1
20160321381 English et al. Nov 2016 A1
20160321817 Ratcliff et al. Nov 2016 A1
20160334230 Ross et al. Nov 2016 A1
20160342837 Hong et al. Nov 2016 A1
20160347322 Clarke et al. Dec 2016 A1
20160375907 Erban Dec 2016 A1
20170053169 Cuban et al. Feb 2017 A1
20170061632 Lindner et al. Mar 2017 A1
20170124476 Levinson et al. May 2017 A1
20170134631 Zhao et al. May 2017 A1
20170177951 Yang Jun 2017 A1
20170227647 Baik Aug 2017 A1
20170301104 Qian Oct 2017 A1
20170305423 Green Oct 2017 A1
20170318407 Meister et al. Nov 2017 A1
20170363423 Dormody et al. Dec 2017 A1
20180005407 Browning Jan 2018 A1
20180111274 Seaok Apr 2018 A1
20180131924 Jung May 2018 A1
20180149739 Becker May 2018 A1
20180151063 Pun et al. May 2018 A1
20180158197 Dasgupta et al. Jun 2018 A1
20180188043 Chen Jul 2018 A1
20180216943 Hawkins Aug 2018 A1
20180260956 Huang et al. Sep 2018 A1
20180268566 Houts Sep 2018 A1
20180283892 Behrendt Oct 2018 A1
20180284278 Russell Oct 2018 A1
20180312125 Jung Nov 2018 A1
20180315201 Cameron Nov 2018 A1
20180364717 Douillard Dec 2018 A1
20180373254 Song Dec 2018 A1
20180373980 Huval Dec 2018 A1
20190025853 Julian Jan 2019 A1
20190063945 Liu Feb 2019 A1
20190065863 Luo et al. Feb 2019 A1
20190066329 Luo et al. Feb 2019 A1
20190066330 Luo et al. Feb 2019 A1
20190108384 Wang et al. Apr 2019 A1
20190132391 Thomas et al. May 2019 A1
20190132392 Liu et al. May 2019 A1
20190163989 Guo et al. May 2019 A1
20190210564 Han et al. Jul 2019 A1
20190210613 Sun et al. Jul 2019 A1
20190226851 Nicosevici Jul 2019 A1
20190236950 Li et al. Aug 2019 A1
20190266420 Ge et al. Aug 2019 A1
20190271549 Zhang et al. Sep 2019 A1
20190312993 Yamashita Oct 2019 A1
20190339084 Korenaga Nov 2019 A1
20200089973 Efland Mar 2020 A1
20200271473 Wang et al. Aug 2020 A1
20210183099 Fujii et al. Jun 2021 A1
20210373161 Lu et al. Dec 2021 A1
Foreign Referenced Citations (51)
Number Date Country
102815305 Dec 2012 CN
105667518 Jun 2016 CN
105825173 Aug 2016 CN
106340197 Jan 2017 CN
106781591 May 2017 CN
106909876 Jun 2017 CN
107111742 Aug 2017 CN
108010360 May 2018 CN
111256693 Jun 2020 CN
2608513 Sep 1977 DE
0890470 Jan 1999 EP
1754179 Feb 2007 EP
2448251 May 2012 EP
2463843 Jun 2012 EP
2761249 Aug 2014 EP
2918974 Sep 2015 EP
2946336 Nov 2015 EP
2993654 Mar 2016 EP
3081419 Oct 2016 EP
3819673 May 2021 EP
10-2017-198566 Nov 2017 JP
100802511 Feb 2008 KR
20170065083 Jun 2017 KR
1991009375 Jun 1991 WO
2005098739 Oct 2005 WO
2005098751 Oct 2005 WO
2005098782 Oct 2005 WO
2010109419 Sep 2010 WO
2013045612 Apr 2013 WO
2014111814 Jul 2014 WO
2014166245 Oct 2014 WO
2014201324 Dec 2014 WO
2015083009 Jun 2015 WO
2015103159 Jul 2015 WO
2015125022 Aug 2015 WO
2015186002 Dec 2015 WO
2016090282 Jun 2016 WO
2016135736 Sep 2016 WO
2017013875 Jan 2017 WO
2017079349 May 2017 WO
2017079460 May 2017 WO
2018132608 Jul 2018 WO
2019040800 Feb 2019 WO
2019084491 May 2019 WO
2019084494 May 2019 WO
2019140277 Jul 2019 WO
2019161134 Aug 2019 WO
2019168986 Sep 2019 WO
2020038118 Feb 2020 WO
2020097512 May 2020 WO
2021017213 Feb 2021 WO
Non-Patent Literature Citations (81)
Entry
Chinese Patent Office, First Search Report for CN 201980013350.2, dated Feb. 21, 2022, 3 pages with machine translation.
Chinese Patent Office, First Office Action for CN 201980013350.2, dated Feb. 25, 2022, 20 pages with machine translation.
Extended European Search Report for European Patent Application No. 18849237.5, dated Apr. 23, 2021.
Extended European Search Report for European Patent Application No. 21166828.0, dated Aug. 5, 2021 (8 pages).
Nguyen, Trung Bao, “Evaluation of Lane Detection Algorithms based on an Embedded Platform,” Jun. 30, 2017, Chapters 3 and 4, pp. 1-107; Master Thesis, Technische Universitat Chemnitz, Jun. 2017, available at https://nbnresolving.org/urn:nbn:de:bsz:ch1-qucosa-226615.
Niu, J., et al., “Robust Lane Detection using Two-stage Feature Extraction with Curve Fitting”, Pattern Recognition, Elsevier, 59, pp. 225-233 (2016).
Narote, S., et al., “A review of recent advances in lane detection and departure warning system”, Pattern Recognition, Elsevier, 73, pp. 216-234 (2018).
Harry Y. Oh, U.S. Appl. No. 15/896,077, Non-Final Office Action dated Mar. 13, 2020, pp. 1-21.
Harry Y. Oh, U.S. Appl. No. 15/896,077, Final Office Action dated Jul. 9, 2020, pp. 1-30.
Harry Y. Oh, U.S. Appl. No. 15/896,077, Non-Final Office Action dated Oct. 1, 2020, pp. 1-34.
Harry Y. Oh, U.S. Appl. No. 16/184,926, Non-Final Office Action dated Oct. 5, 2020, pp. 1-17.
International Application No. PCT/US2019/060547 International Search Report and Written Opinion, dated Jun. 25, 2020, pp. 1-23.
Carle, Patrick J.F., “Global Rover Localization by Matching Lidar and Orbital 3D Maps.”, IEEE, Anchorage Convention Distriction, pp. 1-6, May 3-8, 2010. (Anchorage Alaska, US), May 3-8, 2019.
Young, Lee W., International Application No. PCT/US19/18113 Search Report and Written Opinion dated May 8, 2019. (pp. 1-13).
Caselitz, T. et al., “Monocular camera localization in 3D LiDAR maps,” European Conference on Computer Vision (2014) Computer Vision—ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol. 8690. Springer, Cham.
Mur-Artal, R. et al., “ORB-SLAM: A Versatile and Accurate Monocular SLAM System,” IEEE Transaction on Robotics, Oct. 2015, pp. 1147-1163, vol. 31, No. 5, Spain.
Sattler, T. et al., “Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?” CVPR, IEEE, 2017, pp. 1-10.
Engel J., et al., LSD-SLAM: Large-Scale Direct Monocular SLAM. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds) Computer Vision—ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol. 8690. Springer, Cham.
Levinson, Jesse et al., Experimental Robotics, Unsupervised Calibration for Multi-Beam Lasers, pp. 179-194, 12th Ed., Oussama Khatib, Vijay Kumar, Gaurav Sukhatme (Eds.) Springer-Verlag Berlin Heidelberg 2014.
International Application No. PCT/US2019/013322, International Search Report and Written Opinion dated Apr. 2, 2019.
International Application No. PCT/US19/12934, International Search Report and Written Opinion dated Apr. 29, 2019.
International Application No. PCT/US18/53795, International Search Report and Written Opinion dated Dec. 31, 2018.
International Application No. PCT/US18/57484, International Search Report and Written Opinion dated Jan. 7, 2019.
International Application No. PCT/US2018/057851, International Search Report and Written Opinion dated Feb. 1, 2019.
International Application No. PCT/US2019/019839, International Search Report and Written Opinion dated May 23, 2019.
International Application No. PCT/US19/25995, International Search Report and Written Opinion dated Jul. 9, 2019.
Geiger, Andreas et al., “Automatic Camera and Range Sensor Calibration using a single Shot”, Robotics and Automation (ICRA), pp. 1-8, 2012 IEEE International Conference.
Zhang, Z. et al. A Flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence (vol. 22 , Issue: 11 , Nov. 2000).
International Application No. PCT/US2018/047830, International Search Report and Written Opinion dated Apr. 27, 2017.
Bar-Hillel, Aharon et al. “Recent progress in road and lane detection: a survey.” Machine Vision and Applications 25 (2011): 727-745.
Schindler, Andreas et al. “Generation of high precision digital maps using circular arc splines,” 2012 IEEE Intelligent Vehicles Symposium, Alcala de Henares, 2012, pp. 246-251. doi: 10.1109/IVS.2012.6232124.
International Application No. PCT/US2018/047608, International Search Report and Written Opinion dated Dec. 28, 2018.
Hou, Xiaodi and Zhang, Liqing, “Saliency Detection: A Spectral Residual Approach”, Computer Vision and Pattern Recognition, CVPR'07—IEEE Conference, pp. 1-8, 2007.
Hou, Xiaodi and Harel, Jonathan and Koch, Christof, “Image Signature: Highlighting Sparse Salient Regions”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, No. 1, pp. 194-201, 2012.
Hou, Xiaodi and Zhang, Liqing, “Dynamic Visual Attention: Searching for Coding Length Increments”, Advances in Neural Information Processing Systems, vol. 21, pp. 681-688, 2008.
Li, Yin and Hou, Xiaodi and Koch, Christof and Rehg, James M. and Yuille, Alan L., “The Secrets of Salient Object Segmentation”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 280-287, 2014.
Zhou, Bolei and Hou, Xiaodi and Zhang, Liqing, “A Phase Discrepancy Analysis of Object Motion”, Asian Conference on Computer Vision, pp. 225-238, Springer Berlin Heidelberg, 2010.
Hou, Xiaodi and Yuille, Alan and Koch, Christof, “Boundary Detection Benchmarking: Beyond F-Measures”, Computer Vision and Pattern Recognition, CVPR'13, vol. 2013, pp. 1-8, IEEE, 2013.
Hou, Xiaodi and Zhang, Liqing, “Color Conceptualization”, Proceedings of the 15th ACM International Conference on Multimedia, pp. 265-268, ACM, 2007.
Hou, Xiaodi and Zhang, Liqing, “Thumbnail Generation Based on Global Saliency”, Advances in Cognitive Neurodynamics, ICCN 2007, pp. 999-1003, Springer Netherlands, 2008.
Hou, Xiaodi and Yuille, Alan and Koch, Christof, “A Meta-Theory of Boundary Detection Benchmarks”, arXiv preprint ar Xiv: 1302.5985, 2013.
Li, Yanghao and Wang, Naiyan and Shi, Jianping and Liu, Jiaying and Hou, Xiaodi, “Revisiting Batch Normalization for Practical Domain Adaptation”, arXiv preprint arXiv:1603.04779, 2016.
Li, Yanghao and Wang, Naiyan and Liu, Jiaying and Hou, Xiaodi, “Demystifying Neural Style Transfer”, arXiv preprint arXiv: 1701.01036, 2017.
Hou, Xiaodi and Zhang, Liqing, “A Time-Dependent Model of Information Capacity of Visual Attention”, International Conference on Neural Information Processing, pp. 127-136, Springer Berlin Heidelberg, 2006.
Wang, Panqu and Chen, Pengfei and Yuan, Ye and Liu, Ding and Huang, Zehua and Hou, Xiaodi and Cottrell, Garrison, “Understanding Convolution for Semantic Segmentation”, arXiv preprint arXiv:1702.08502, 2017.
Li, Yanghao and Wang, Naiyan and Liu, Jiaying and Hou, Xiaodi, “Factorized Bilinear Models for Image Recognition”, arXiv preprint arXiv:1611.05709, 2016.
Hou, Xiaodi, “Computational Modeling and Psychophysics in Low and Mid-Level Vision”, California Institute of Technology, 2014.
Spinello, Luciano, Triebel, Rudolph, Siegwart, Roland, “Multiclass Multimodal Detection and Tracking in Urban Environments”, Sage Journals, vol. 29 Issue 12, pp. 1498-1515 Article first published online: Oct. 7, 2010; Issue published: Oct. 1, 2010.
Matthew Barth, Carrie Malcolm, Theodore Younglove, and Nicole Hill, “Recent Validation Efforts for a Comprehensive Modal Emissions Model”, Transportation Research Record 1750, Paper No. 01-0326, College of Engineering, Center for Environmental Research and Technology, University of California, Riverside, CA 92521, date unknown.
Kyoungho Ahn, Hesham Rakha, “The Effects of Route Choice Decisions on Vehicle Energy Consumption and Emissions”, Virginia Tech Transportation Institute, Blacksburg, VA 24061, date unknown.
Ramos, Sebastian, Gehrig, Stefan, Pinggera, Peter, Franke, Uwe, Rother, Carsten, “Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling”, arXiv:1612.06573v1 [cs.CV] Dec. 20, 2016.
Schroff, Florian, Dmitry Kalenichenko, James Philbin, (Google), “FaceNet: A Unified Embedding for Face Recognition and Clustering”, CVPR 2015.
Dai, Jifeng, Kaiming He, Jian Sun, (Microsoft Research), “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, CVPR 2016.
Huval, Brody, Tao Wang, Sameep Tandon, Jeff Kiske, Will Song, Joel Pazhayampallil, Mykhaylo Andriluka, Pranav Rajpurkar, Toki Migimatsu, Royce Cheng-Yue, Fernando Mujica, Adam Coates, Andrew Y. Ng, “An Empirical Evaluation of Deep Learning on Highway Driving”, arXiv:1504.01716v3 [cs.RO] Apr. 17, 2015.
Tian Li, “Proposal Free Instance Segmentation Based on Instance-aware Metric”, Department of Computer Science, Cranberry-Lemon University, Pittsburgh, PA., date unknown.
Mohammad Norouzi, David J. Fleet, Ruslan Salakhutdinov, “Hamming Distance Metric Learning”, Departments of Computer Science and Statistics, University of Toronto, date unknown.
Jain, Suyong Dutt, Grauman, Kristen, “Active Image Segmentation Propagation”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Jun. 2016.
MacAodha, Oisin, Campbell, Neill D.F., Kautz, Jan, Brostow, Gabriel J., “Hierarchical Subquery Evaluation for Active Learning on a Graph”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Kendall, Alex, Gal, Yarin, “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision”, arXiv:1703.04977v1 [cs.CV] Mar. 15, 2017.
Wei, Junqing, John M. Dolan, Bakhtiar Litkhouhi, “A Prediction- and Cost Function-Based Algorithm for Robust Autonomous Freeway Driving”, 2010 IEEE Intelligent Vehicles Symposium, University of California, San Diego, CA, USA, Jun. 21-24, 2010.
Peter Welinder, Steve Branson, Serge Belongie, Pietro Perona, “The Multidimensional Wisdom of Crowds”; pp. 1-8, Vision Caltech.Edu (2010). http://www.vision.caltech.edu/visipedia/papers/WelinderEtaINIPS10.pdf, 2010.
C. Yang, Z. Li, R. Cui and B. Xu, “Neural Network-Based Motion Control of an Underactuated Wheeled Inverted Pendulum Model,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 25, No. 11, pp. 2004-2016, Nov. 2014.
Stephan R. Richter, Vibhav Vineet, Stefan Roth, Vladlen Koltun, “Playing for Data: Ground Truth from Computer Games”, Intel Labs, European Conference on Computer Vision (ECCV), Amsterdam, the Netherlands, 2016.
Thanos Athanasiadis, Phivos Mylonas, Yannis Avrithis, and Stefanos Kollias, “Semantic Image Segmentation and Object Labeling”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, No. 3, Mar. 2007.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding”, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, 2016.
Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee, “DESPOT: Online POMDP Planning with Regularization”, Department of Computer Science, National University of Singapore, date unknown.
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. CoRR, abs/1606.02147, 2016.
Szeliski, Richard, “Computer Vision: Algorithms and Applications”, pp. 1-979. Sep. 3, 2010. http://szeliski.org/Book/2010.
Kyoungho Ahn, Hesham Rakha, “The Effects of Route Choice Decisions on Vehicle Energy Consumption and Emissions”, Virginia Tech Transportation Institute, date unknown.
Office Action Mailed in Chinese Application No. 201810025516.X, dated Sep. 3, 2019.
Luo, Yi et al. U.S. Appl. No. 15/684,389 Notice of Allowance dated Oct. 9, 2019.
Kai Yu, Yang Zhou, Da Li, Zhang, Kaiqi Huang, “Large-scale Distributed Video Parsing and Evaluation Platform”, Center for Research on Intelligent Perception and Computing, Institute of Automation, Chinese Academy of Sciences, China, arXiv:1611.09580v1 [cs.CV] Nov. 29, 2016.
P. Guarneri, G. Rocca and M. Gobbi, “A Neural-Network-Based Model for the Dynamic Simulation of the Tire/ Suspension System While Traversing Road Irregularities,” in IEEE Transactions on Neural Networks, vol. 19, No. 9, pp. 1549-1563, Sep. 2008.
Mingdong Wang et al., U.S. Appl. No. 16/184,926 Notice of Allowance dated Jan. 15, 2021, pp. 1-5.
Siyuan Liu, U.S. Appl. No. 17/074,468 Notice of Allowance dated Oct. 7, 2022, p. 1-7.
Chenzhe Qian, U.S. Appl. No. 17/225,396, Non-Final Office Action dated Oct. 24, 2022, pp. 1-19.
Chinese Patent Office, First Office Action for CN 201880055025.8, dated Dec. 16, 2022, 10 pages.
Schindler, et al., “Generation of High Precision Digitial Maps using Circular Arc Splines,” 2012 Intelligent Vehicles Symposium, Alcala de Henares, Spain, Jun. 3-7, 2012.
Mingdong Wang, U.S. Appl. No. 17/320,888, Non-Final Office Action dated Jan. 18, 2023, pp. 1-8.
Examination Report from corresponding European Patent Application No. 21166828.0, dated Mar. 16, 2023 (8 pages).
Xiao Zhongyang et al: “Monocular Vehicle Self-localization method based on Compact Semantic Map”, (2018 21st International Conference on Intelligent Transportation Systems (ITSC), IEEE, Nov. 4, 2018, pp. 3083-3090.
Related Publications (1)
Number Date Country
20210278232 A1 Sep 2021 US
Continuations (1)
Number Date Country
Parent 15896077 Feb 2018 US
Child 17308803 US