Patent Application
20220157168
The invention relates to systems and methods for short-range locating and identification of vehicles in traffic.
Autonomously operated vehicles are expected to cooperate with each other to avoid traffic hazards and facilitate the flow of traffic generally. Such cooperation relies on knowing the locations of other vehicles in proximity and, if wirelessly connected, identifying their access codes.
What is needed is means for vehicles in traffic to determine the locations and, if connected, the wireless addresses of other proximate vehicles.
This Background is provided to introduce a brief context for the Summary and Detailed Description that follow. This Background is not intended to be an aid in determining the scope of the claimed subject matter nor be viewed as limiting the claimed subject matter to implementations that solve any or all of the disadvantages or problems presented above.
In a first aspect, there is a method for a first vehicle to communicate with a second vehicle, the second vehicle proximate to a third vehicle, the method comprising: broadcasting a planning message specifying a particular time; at the particular time, acquiring a first image depicting the second vehicle and the third vehicle; receiving, from the second vehicle, an imaging message comprising a second image, the second image acquired by the second vehicle at the particular time, the second image depicting the first vehicle and the third vehicle; and determining, according to the first image and the second image, a coordinate listing comprising a position of the first vehicle, a position of the second vehicle, and a position of the third vehicle.
In another aspect, there is non-transitory computer-readable media in a second vehicle, the second vehicle in traffic, the traffic comprising a first vehicle and at least one other vehicle, the media containing instructions that when implemented by a computing environment cause a method to be performed, the method comprising: receiving, from the first vehicle, a planning message specifying a time; acquiring, at the specified time, an image comprising the first vehicle and the at least one other vehicle; transmitting, to the first vehicle, an imaging message comprising the image; and receiving, from the first vehicle, a coordinate listing or a traffic map comprising positions of the first vehicle, the second vehicle, and the at least one other vehicle.
In another aspect, there is a computer containing an artificial intelligence structure comprising; one or more inputs, each input comprising an image of traffic, the traffic comprising a plurality of vehicles; one or more internal functions, each internal function operably linked to one or more of the inputs; and an output operably linked to the one or more of the internal functions, the output comprising a prediction of a two-dimensional position of each vehicle of the plurality.
This Summary is provided to introduce a selection of concepts in a simplified form. The concepts are further described in the Detailed Description section. Elements or steps other than those described in this Summary are possible, and no element or step is necessarily required. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended for use as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
These and other embodiments are described in further detail with reference to the figures and accompanying detailed description as provided below.
Like reference numerals refer to like elements throughout.
Disclosed herein are procedures enabling autonomous and semi-autonomous vehicles in traffic to determine the locations of other vehicles, and of landmarks, by exchange of image data. Systems and methods disclosed herein (the “systems” and “methods”, also occasionally termed “embodiments” or “arrangements”, generally according to present principles) can provide urgently needed wireless communication protocols to provide situational awareness in traffic, localize both wirelessly connected and non-connected vehicles, and communicate specifically (unicast) with selected vehicles. With such capabilities, vehicles can reduce traffic fatalities, facilitate traffic flow, and provide V2V and V2X communication options appropriate for 5G and 6G technologies, according to some embodiments.
Autonomous and semi-autonomous vehicles are potentially able to provide greatly increased vehicle safety and traffic efficiency by detecting other vehicles, determining their locations, and making wireless contact with each connected vehicle. In addition, vehicles may detect non-vehicle entities such as pedestrians, obstructions, traffic lights, and signage, along with items related to communications such as 5G/6G access points. Traffic awareness and V2X connectivity are essential in detecting hazards, planning hazard avoidance strategies, and communicating with the other vehicles to cooperatively react to an imminent collision. By exchanging messages at electronic speeds, computer-operated vehicles can coordinate their actions, cooperatively adjust their speed and direction, and thereby avoid almost all types of highway accidents, saving countless lives.
As used herein, a device or entity “knows” something if the device or entity has the relevant information. An “autonomous” vehicle is a vehicle operated by a processor, with little or no human control most of the time. A “semi-autonomous” vehicle is a vehicle at least partially operated by a processor, or which can be fully operated by a processor temporarily, such as during emergency intervention. A wireless message is “unicast” if it is addressed specifically to a particular recipient, and “broadcast” if it is transmitted without specifying a recipient. A wireless entity can transmit “specifically” if the intended recipient is spatially localized and identified by the transmitting entity, as opposed to transmitting blindly to a wireless address. “V2V” means vehicle-to-vehicle messaging. “V2X” means vehicle-to-everything messaging. A “vehicle” is to be construed broadly, including any mobile conveyance such as cars, trucks, busses, motorcycles, scooters, and the like. “Wireless entities” are systems or devices capable of wireless communication such as connected vehicles, pedestrians with smart phones, roadside access points or base stations, and so forth. “Items” or “objects” in images include things that can appear in images. A “traffic map” is a map showing vehicles and other items in a two-dimensional distribution including wireless addresses of the items when known, whereas a “coordinate listing” is a list of location coordinates of the items, their wireless addresses if known, and other information, associated with the items. Items in a traffic map or coordinate listing may include items viewed or visible by at least one vehicle. Such items may include vehicles, persons, roadside objects such as buildings or signs or distinctive vegetation, or distant objects that can help align images taken from different angles and positions. An image “includes” or “comprises” an object if the image depicts the object. “View” and “visible” are to be construed broadly, including detectable by sensors or instruments, and/or imaged (or able to be imaged) from a viewpoint, in visible or infrared light or other imaging medium. “MAC” (medium access control) and “RNTI” (radio network temporary identification) are wireless address codes. A “demodulation reference” is a series of modulated resource elements configured to exhibit levels (such as amplitude or phase levels) of a modulation scheme, as opposed to data. “AI” (artificial intelligence) is computer-based decision-making, as used herein, usually involving a large number of interacting factors in a complex problem.
Traffic “situation awareness” includes determining which vehicles are present, measuring their locations or distances relative to the measuring entity or other point, and determining their status. “Status” in this context includes whether a vehicle is autonomous, semi-autonomous, or human-controlled, wirelessly connected or not, and the wireless address if known. For detecting other vehicles, many autonomous vehicles include cameras (visible and infrared) and sensors (such as lidar, radar, and the like). However, these imagers and sensors cannot detect a hidden vehicle such as a car obscured by an intervening truck, among many other traffic situations. The traffic map (an image showing positions of vehicles, usually annotated with their wireless addresses), or the corresponding coordinate listing (a list of coordinates of the vehicles and wireless addresses), may reveal items that are hidden to one vehicle but visible to another vehicle.
“Localization” means determining the locations or positions or two-dimensional coordinates of vehicles in traffic, in some coordinate system. If the distances are not known, the coordinates may have an arbitrary distance scale. A traffic map with an arbitrary distance scale is still useful to the vehicles by indicating which of the vehicles has which wireless address. For example, the traffic map can inform a particular vehicle that another vehicle directly in front is autonomous and has a certain wireless address, so that the particular vehicle can then communicate with the vehicle in front for cooperation in hazard avoidance. Localization using satellite-based navigation systems such as GPS generally cannot reliably resolve closely-spaced vehicles such as vehicles in adjacent lanes, because the motion of the vehicles can distort the satellite signals. In addition, GPS depends on correlating signals from multiple satellites, and the distance traveled by the vehicle between acquisitions from the various satellites can result in further uncertainty. In addition, satellite coverage is spotty in many urban environments, steep canyons, and many other places where traffic situation awareness is needed. Localization using narrow directed beams of radio waves is also unlikely to provide the necessary reliability because large phased-array antennas are needed for degree-level spatial resolution. In addition, high-frequency beams are notoriously prone to reflections from nearby conducting objects, such as vehicles in dense traffic, further confusing the single-vehicle selectivity. In addition, the beam necessarily continues propagating beyond the intended recipient and can be received by other entities in line. Therefore, except for the simplest idealized cases, situation awareness of dense traffic at freeway speeds remains an unsolved problem, absent the systems and methods disclosed below.
As detailed in the examples, vehicles can obtain real-time traffic awareness by exchanging images from their various viewpoints, and then combining the images (that is, perform “viewpoint fusion”) to generate a traffic map of the vehicles and/or a coordinate listing of the two-dimensional coordinates of the vehicle positions. The vehicles can exchange their wireless addresses at the same time, so that the traffic map or coordinate listing can associate each vehicle's wireless address with its position. A “planning entity” is a vehicle or other wireless entity that initiates the procedure by broadcasting a “planning message”, requesting that other vehicles take an image, such as a 360-degree image, of surrounding traffic at a particular time. The “participating vehicles” acquire their images of traffic from their own viewpoints at the specified time, and then transmit the image in an “imaging message” to the planning entity for analysis. The planning entity then combines or “fuses” the images with its own image, also acquired at the specified time, and thereby produces a coordinate listing of the vehicle coordinates. The planning entity can also make a traffic map depicting the vehicles in relative positions from each other. The coordinate listing and traffic map may have an arbitrary distance scale, or they may have a calibrated distance scale according to a measured distance between, for example, the planning entity and one of the vehicles. The planning entity can then broadcast the coordinate listing and/or the traffic map in a “mapping message”. The other vehicles can receive the mapping message, determine the wireless address of a selected vehicle, and communicate specifically with that vehicle.
To determine the overall distance scale of the traffic map or listing, the planning entity can measure the distance to at least one vehicle using a sensor, such as radar or lidar, and calibrate all the other distances to that measurement. Optionally, the participating vehicles can measure distances to some of the items in their view, and can include those distances in their imaging messages. The planning entity may be configured to include, in the coordinate listing or map, the distance values so determined. In addition, many coordinate values may be over-determined by multiple vehicle viewpoints, and the planning entity may resolve disagreements between corresponding distance measurements by averaging or least-squares analysis or maximum likelihood fitting or other analysis calculation suitable for combining multiple measurements.
The planning entity can include its wireless address in the planning message, and the participating vehicles can include their wireless addresses in their imaging messages. After receiving one or more imaging messages from the participating vehicles, the planning entity can perform viewpoint fusion by identifying objects that appear in two or more of the images The planning entity can then, by triangulation, determine the two-dimensional coordinates of each visible item. (However, if the item and the two viewpoints happen to lie on a straight line, a third non-collinear viewpoint may be needed to disambiguate the location.) The traffic map or listing may include the coordinates of each of the participating vehicles and their wireless addresses, so that the vehicles can then use the traffic map to determine which wireless address belongs to which vehicle in their view. The traffic map or listing can also include non-participating vehicles that appear in the images. The traffic map or listing can thereby provide the participating vehicles, and other entities receiving the traffic map, full traffic situation awareness. In addition, by providing the wireless addresses of each vehicle if known, the traffic map or listing can enable the vehicles (and wireless fixed assets) to specifically communicate with each other, and thereby cooperate to avoid traffic hazards and improve traffic management.
The planning and imaging messages may also include subsidiary information, such as the GPS coordinates of the transmitting vehicle if known, as well as the transmitting vehicle type (sedan, bus, truck, etc.), the color, and/or which lane that the transmitting vehicle is in, among other parameters which may simplify the viewpoint fusion and resolve ambiguities. If the GPS data is included, preferably the GPS acquisition time and the speed of the vehicle are also indicated, so that the planning entity can correct for the displacement of the vehicle between the time of GPS acquisition and image acquisition.
In viewpoint fusion, the planning entity may analyze multiple images of the same, or nearly the same, vehicles from different viewpoints. The planning entity can calculate a mutually consistent location for each of the items in view as well as the locations of the vehicles acquiring the images. However, each of those overlapping location determinations is likely to be slightly different, due to measurement uncertainties and limited resolution achievable. The planning entity can combine the various measurements with a fitting function, such as maximum-likelihood, or least-squares, or mean with outlier-rejection, or other type of function to determine a best-fit location for each item in the traffic map or listing. In this way, items that are viewed by multiple participating vehicles may be located with higher precision than achievable by each of the observers individually. In addition, the locations of the vehicles acquiring the images can be determined as part of the self-consistent spatial distribution solution.
The vehicles may transmit their messages according to 5G or 6G sidelink specifications. If an access point or base station is within range, the communication may be according to “sidelink mode-1” in which the base station sets the timing and manages the vehicle messages. If no network interface is available, the vehicles themselves may set up the timing and bandwidths of a “sidelink mode-2” local network. Alternatively, the vehicles may transmit their messages at-will or according to another wireless technology such as Wi-Fi.
The planning entity may use artificial intelligence or machine learning in several steps of this procedure. AI may be useful for determining the type of each object that is viewed from different directions. AI may also be useful for determining the self-consistent distribution of item locations based on the images, Finally, AI may assist in combining redundant position determinations by, for example, recognizing and rejecting outliers.
Some vehicles in traffic may not be visible to other vehicles, due to an intervening item such as a truck. However, a vehicle that is hidden from a first vehicle may be visible to another vehicle. In that case, the hidden vehicle can be placed on the traffic map based on the images that include it. Then, upon receiving the traffic map, the first vehicle may discover the existence and location of that hidden vehicle. Revealing hidden vehicles or pedestrians could be life-saving. For example, if the first vehicle detects an imminent collision, it may ask the truck to change lanes. But if there is a hidden vehicle beside the truck, then the truck would refuse to change lanes, and precious time would be lost while the first vehicle searches for another mitigation. If, however, the first vehicle knows that the hidden vehicle is present, the first vehicle can select a different avoidance strategy without delay.
Optionally, each participating entity can mark, on its traffic image, its direction of travel, for example by placing an icon on the image corresponding to the front of the vehicle or to a line passing centrally and longitudinally through the vehicle. If the road is straight, then each of the participating vehicles generally share the same travel direction, aside from lane changes and the like, and can use the road direction as a common angle reference. Alternatively, a vehicle may have a compass such as an electronic compass that indicates what direction is north, which can be the angle reference. Each entity can add an icon or other indication to its image, indicating the travel direction (which is the same as the road direction unless changing lanes), or north or other specified geographic direction. Alternatively, a code may be included with the imaging message indicating which column of pixels in the image is aligned with the travel direction or other geographical direction, so that the planning entity can align the images from multiple participating vehicles accordingly. Although images with random orientations can be fused by rotating them until they come into alignment according to the objects appearing in both images, image fusion is much simpler if the images share a common orientation.
In some embodiments, the planning entity may include software configured to recognize vehicles, or other objects, viewed from different directions. For example, a pickup truck looks very different when viewed from the side or front or back. Image processing software may assign a model shape, such as a generic three-dimensional pickup truck shape, to such an item in the image, and may thereby recognize the vehicle from different directions in the other images.
In summary, the planning entity may transmit a planning message to other participating vehicles, specifying a time to synchronize their image acquisition. The participating vehicles (and other entities in range) may then acquire their traffic images at that time, and transmit each image in an imaging message to the planning entity. The imaging message may also include the participating vehicle's wireless address code and other identifying information. The planning entity can then combine the various images, and optionally its own image of the traffic, and thereby prepare a traffic map or listing of the positions of the various vehicles and entities in view. In addition, the traffic map or listing may include the wireless addresses of the vehicles, if known. The planning entity can then broadcast a mapping message that includes the traffic map or coordinate listing. Each of the participating vehicles, and other entities, can receive the mapping message, identify its own position in the traffic map or listing according to its wireless address, and thereby determine which of the vehicles in its view are associated with which wireless addresses. The entities can then communicate specifically with the other vehicles, and thereby cooperate in avoiding hazards and managing the flow of traffic.
The following examples illustrate a process for acquiring and combining traffic images from multiple viewpoints.
In the depicted example, the first vehicle 101 broadcasts a wireless planning message requesting that other vehicles in range acquire images, such as 360-degree images, of the traffic in view, and that the image acquisition be at a particular time, such as 100 milliseconds after the planning message. The planning message may also request that the images be transmitted back to the first vehicle 101 using a predetermined format. The first vehicle 101 can receive the imaging message and perform viewpoint fusion by correlating and triangulating items visible in both images.
The boxes for the third and fourth vehicles 133, 134 are shown dashed to indicate that those vehicles are present but did not respond to wireless messages and are presumed to not be in wireless communication. The first and second vehicle boxes 131, 132 are shown bold because they participated by providing images from which the traffic map 130 was derived. The box representing the fifth and sixth items 136, 136 are shown solid but not bolded because they are in radio contact, but did not provide images. Wireless addresses are provided in three of the boxes 131, 132, 136. The first vehicle 101 included its wireless address in the planning message, and the second vehicle 102 included its wireless addresses in the imaging message. The roadside access point 106 provided its wireless address in a separate message. The fifth vehicle 105 with a blank box 135 is wirelessly connected but not participating; hence there is not yet enough information to correlate its wireless address with its physical location. After receiving the traffic map (or listing), the fifth vehicle 105 noticed that the box 135 representing its location is blank, and then may transmit a location message indicating that it is the vehicle represented by the box 135, while providing its wireless address, so that each of the vehicles may then update their traffic maps or listings to include that address.
Initially, vehicle-1 broadcasts a planning message 201 suggesting that all vehicles in range acquire images of traffic at a particular specified time 202. Three vehicles do so (one image acquisition is labeled as 203). The vehicles then wait a random LBT (listen-before-talk) interval 204 (arrow here and elsewhere) before transmitting their imaging messages. After each LBT interval, the vehicles determine whether another vehicle is transmitting, and if so, the ready vehicle waits another random interval after the current transmission is finished. In this case, vehicle-1 happens to have selected the shortest random LBT interval 204. Vehicle-1 then (detecting no interfering transmissions) broadcasts its imaging message 205 as shown.
Vehicles-2 and -3 expire their waiting intervals and are then ready to transmit. However, they detect the vehicle-limaging message 205 when their LBT intervals expire, and therefore they abort their own transmissions as indicated by slashed boxes (one labeled as 206). After the vehicle-1 imaging message is done, vehicles-2 and -3 again delay random LBT intervals. Vehicle-2 wins this time, and broadcasts its imaging message 207. After message 207 is done, vehicle-3 waits another LBT interval and broadcasts its imaging message 208. The imaging messages 205, 207, and 208 may include additional information, such as the wireless address of the transmitting entity, measured distances between the transmitting entity and various items in view, a time offset at which the image was acquired if not at the specified time 202, the velocity and/or acceleration of the transmitting entity, whether the transmitting entity is a vehicle or pedestrian or fixed asset or whatever, whether the transmitting entity is computer-controlled or human-driver-operated, GPS coordinates, lane occupation, and other optional information, according to some embodiments. (In another embodiment, the vehicles may have been assigned different frequency bands, and may be able to transmit their imaging messages simultaneously on different frequencies, thereby avoiding message interference and saving time.)
In some embodiments, various entities may analyze the imaging messages, identify items in the images, correlate multiple views of the items, and thereby perform viewpoint fusion to develop a traffic map or listing. In the depicted case, the entity that initially broadcast the planning message 201, which is vehicle-1, performs the analyses at 209, and then broadcasts the resulting traffic map or listing 210 indicating the two-dimensional locations of items seen in the imaging messages. The traffic map or listing may include item coordinates relative to vehicle-1 or other coordinate origin. The other vehicles can determine distances from their own viewpoint by subtraction, shifting the origin from vehicle-1 to, say, vehicle-2 or vehicle-3. In addition, each item listed in the traffic map may show, or refer to, additional information about that item, such as its wireless address, an indication of the class of item (vehicle, roadside object, etc.), the type of vehicle (car, semi, bus, pickup, motorcycle, etc.), and other information about each item in the traffic map or listing.
In some embodiments, other entities, besides the initial planning entity, may process the images and generate their own traffic map or listing. For example, vehicle-2 may perform the analyses and transmit its version of the traffic map 211 (in dash) instead of vehicle-1, or at a separate time, or on a separate frequency. Alternatively, the access point 106 may receive the imaging messages 205, 207, and 208, and may prepare its own version of the traffic map, and broadcast it at 212. Since the computing power of an access point is generally superior to that of most vehicles, the access point may be able to prepare the traffic map sooner than the vehicles, or with better accuracy, or other advantages. Whether one or more of the vehicles prepares the traffic map, or the access point or other entity does so, may depend on factors such as whether an access point is available within range of the low-power transmissions of the vehicles. In some embodiments, the planning entity determines the traffic map by default, but may withhold its map message if another entity, such as the access point, transmits its map first. Optionally, the various entities may indicate, in broadcast messages for example, which entities will prepare traffic maps.
Transmitting large images wirelessly tends to occupy significant time, and in the envisioned application, multiple vehicles may be waiting to transmit their image data. Therefore, the imaging messages may be encoded for speed and compactness. For example, pixels may be merged since high resolution is usually not necessary for determining vehicle locations. The image pixels may be rendered in black-gray-white instead of full color. The images are expected to provide enough shape and texture information to enable each object to be identified in multiple viewpoints, and sufficient spatial information to resolve adjacent vehicles reliably. As a further compactification, the image may be divided into sections of different spatial resolution, with higher pixel resolution in the sections that include vehicles, and low resolution in sections showing the sky. Further image compactification may be arranged using methods known to those with skill in the digital imaging arts.
At 304, the first vehicle (and optionally other entities) receives the imaging messages from the participating vehicles. The first vehicle analyzes the received images along with its own image acquired at the specified time, and identifies objects that appear in more than one of the images. At 305, the first vehicle calculates the locations of those objects in a two-dimensional coordinate system, thereby determining a listing of coordinates and, optionally, a traffic map displaying those objects. The coordinate axes may be geographical (such as latitude and longitude lines) or local (such as parallel and perpendicular to the roadway), for example. When the vehicle locations are dependent on multiple viewpoints with unknown measurement errors, the first vehicle may perform a fitting routine to determine each vehicle position using, for example, a least-squares or maximum-likelihood or other best-fit location for each vehicle. The first vehicle may select objects to include in the traffic map according to relevance. For example, the traffic map may include vehicles and pedestrians but not trees or buildings (although those fixed objects may be useful in analyzing the various viewpoints). In addition, AI may be useful in each of those tasks.
At 306, the first vehicle determines which one, of the objects in the traffic map or coordinate listing, is associated with a known wireless address. The first vehicle then specifies the associated wireless address along with the location of the object in the map and listing. For example, the first vehicle may determine an object's wireless address from an imaging message transmitted by the object, and may include, in the map data associated with the object, the wireless address of the object. The mapping message may include a coordinate listing of two-dimensional coordinates of the objects in the map, relative to a common origin such as the location of the first vehicle, and using a common coordinate system such as the direction of the roadway. The first vehicle may also include, in the mapping message, in a section providing data about each object in the map, a code indicating the type of object, such as car, bus, truck, pedestrian, motorcycle, bicycle, fixed asset, and so forth.
At 307, the first vehicle broadcasts a mapping message, including the local traffic map or coordinate listing, to other proximate vehicles. The mapping message may be a two-dimensional image such as an overhead view of the local traffic. Alternatively, and more compactly, the mapping message may be formatted as a list of coordinates specifying the location of each (relevant) object in view. Each object may be a vehicle or pedestrian or other entity, especially wireless or mobile objects, in the images. If the distance scale is known, the coordinates may be in meters relative to an origin, such as an origin at the first vehicle or other relevant point. If the distance scale is not known, the map and listing may provide relative locations in an arbitrary scale, such that the vehicles involved can determine which other vehicles are around them even if the separation distance is not specified. It is generally sufficient for the first vehicle to measure one separation distance, using radar or lidar for example, and then to calibrate all the other distances and coordinates accordingly. In the coordinate system, one of the axes (say the X axis) may be parallel to the roadway (at the position of the first vehicle) and the other coordinate (Y) may be perpendicular to the roadway. Other configurations of the map data are possible and foreseen.
At 308, each vehicle, and other wireless entities in range, can receive the mapping message and determine where the other vehicles are located. Vehicles that are hidden to some of the participating vehicles, but are in view of other participating vehicles, can be revealed in the map or listing. The wireless addresses of each participating, or otherwise known, entity may be included in the map or listing, so that the entities can communicate specifically with each other. For example, a vehicle detecting an imminent collision with a vehicle in front may determine, from the map or listing, the wireless address of that vehicle in front, and may transmit an emergency message specifically to the other vehicle instructing it to take immediate evasive action. Absent the procedures disclosed, the vehicles would not know how to contact a specific other vehicle, and cooperative collision avoidance would be vastly more difficult.
The following examples illustrate possible message formats for the planning, imaging, and mapping messages.
Vehicles receiving the mapping message may determine which of the vehicles in surrounding traffic have which wireless addresses, and may therefore contact one of the other vehicles selectively and communicate specifically with the selected vehicle. Absent this ability, cooperation for collision avoidance and traffic management would be far more difficult.
The systems and methods further include artificial intelligence (AI) models with machine learning (ML), specifically to assist in the viewpoint fusion process, as detailed in the following examples.
An embodiment of AI for viewpoint fusion pertains to object recognition in images. A pickup truck, for example, looks very different when viewed from the front, side, and back. Other items, such as two white cars of a particular make and model, may look quite similar. Distinguishing similar-looking items while recognizing an item viewed from different directions, is a complex problem. However, it is generally not possible to do viewpoint fusion until specific items are identified in at least two of the images (or, if present in only one image, the distance may be measured and provided along with the image). Therefore, the systems and methods include an AI model configured to take, as input, multiple images of a set of vehicles viewed from various directions, and to produce, as output, an indication of which items in the images correspond to each other. For example, the AI model may include a database of vehicle prototypes, each vehicle prototype including a three-dimensional shape with characteristic features such as windows of a particular shape, bumpers of a particular design, trim, lights, etc. By comparing the imaged items to the vehicle prototypes, as the prototypes would appear from various angles, the AI model may determine whether each item in each image corresponds to the same prototype, and therefore to the same physical object.
In addition, the AI model may be configured to distinguish two similar-looking items according to subtle differences in detail, such as a mud splash appearing on one of the images but not the other, a difference in the number of occupants, or feature differences such as the hubcap design or a bumper sticker or the license plate code (if visible), for example. To do so, the AI model may be configured to start with the closest relevant prototype, then populate the prototype with features visible in one of the images, and determine whether those features are present in the other images. If so, they are probably all the same physical object. If they differ in even one discrete feature that should be visible based on the viewpoints, the images probably depict two different, but similar, objects.
The AI model may be based on an AI structure such as a neural net, in a computer such as a supercomputer. The AI structure may have multiple inputs such as the images acquired at multiple viewpoints, and multiple outputs such as a coordinate listing indicating locations of each item appearing in more than one image, and a number of internal functions that include adjustable variables. The internal functions may take input data, such as the images, and may process the images by digitization, feature parsing, statistical correlation, and other image processing techniques to extract features. The internal functions may pass processed data to other internal functions, which may pass further processed data to the output. The output may include a listing of image items identified as vehicles or other items, and their positions.
The output coordinate listing may then be compared to the actual distribution of vehicles, as determined by an overhead photograph or by human analysts for example, thereby determining the accuracy of the AI prediction. The adjustable variables may then be varied, singly or in groups, to improve the accuracy of the image-object assignments and location determinations. For example, one variable may be a weighting factor or threshold, determining how similar (or different) two images can be while depicting the same object. Such a weighting factor may play an important role in separating two different similarly-shaped objects, versus a single object viewed from two different directions. Many other variables in the internal functions may have more obscure meanings or none at all. Nevertheless, if the AI structure has a sufficient number of layers and adjustable variables, when “trained” using a sufficiently large number of traffic image examples, the AI model is (usually) able to discern the vehicles, and identify them among the various images, and predict their relative positions, with satisfactory accuracy for most applications.
Due to the potentially large number of inputs and adjustable variables in the model, and the very large amount of training data likely needed for convergence of the model, the AI structure is preferably prepared in a supercomputer. The supercomputer may be a classical semiconductor-based computer, with sufficient speed and thread count and processor count to perform the model training in a feasible amount of time. Alternatively, the supercomputer may be a quantum computer having “qbits” or quantum bits as its working elements. Quantum computers may provide especial advantages to solving AI models because they can very rapidly explore a complex terrain of values, such as the highly interrelated effects of the various inputs on the output results. Therefore, the systems and methods include a quantum computer programmed to include an AI structure and trained on images acquired by each of several vehicles in traffic, and to prepare a comprehensive map of the vehicles, and optionally fixed items, detected by the vehicles.
The internal functions 503, 505 perform mathematical and logical operations on the data provided to each internal function by the links 502, 504 leading to each internal function, such as weighted averaging, nonlinear operations such as range compression, logical operation such as selecting one of the links based on the data from another link, among many other possible operations. The internal functions include adjustable variables that can be adjusted or “trained” to solve a particular problem, such as image viewpoint fusion to generate a two-dimensional map. The links 502, 504 may also include operations such as weighting, in some embodiments. Some embodiments include feedback and more complex topologies, as indicated by the dashed arrow 514. The results of the last layer 505 of internal functions is then conveyed 506 to the output 507, which may calculate an average, or a maximum, or other combination of the processed data provided to it by the last set of links 506. Although the diagram shows links connecting just a few of the functions in each layer, in an actual AI structure each function of one layer may link to all of the functions of the next layer. Only two layers are shown, but larger numbers of layers of internal functions are generally required to obtain satisfactory predictions.
The output 507 is a traffic map or coordinate listing. Also shown is an extra input 511 which is the “ground truth”, the actual distribution of vehicles in traffic, as determined by an overhead camera or other means for determining the two-dimensional distribution of the vehicles independently of the input images. This extra input 511 is not provided to the structure 500, but rather is used as a training tool by comparing 513 the actual coordinate listing 512 with the output 507 of predicted locations, and thereby determining an accuracy of the output 507.
The AI structure 500 can be “tuned” or “trained”, thereby forming an AI model of the traffic imaging application, by adjusting the internal variables until the output coordinate listing is sufficiently accurate. For example, the variables can be adjusted singly or in groups, in various directions and by various amounts, to determine whether any of those variations results in better agreement between the output 507 and the actual traffic distribution 512. If so, the variables can be adjusted further in the same manner, and if not, the variables can be reversed or adjusted in some other way. In millions or billions (or trillions) of iterative variations, the predictive accuracy may incrementally improve, and the computer may retain the best-performing variable set so far obtained. The computer may start multiple separate explorations of different configurations of the variables, following each separately to search for the best prediction accuracy. By repeating this process for many traffic scenarios, if the AI structure 500 is sufficiently broad and versatile, the output 507 may successively approach the actual distributions 512, in most situations. The AI model is then “trained”.
Typically AI structures are large and unwieldy, impractical for field use by traveling vehicles. Therefore, a fieldable algorithm may be developed from the AI model. Typically, some or many of the links and internal functions may have little effect on the output. Therefore, a simpler and more compact version of the AI model may be developed by “pruning” or deleting the unproductive links, inputs, and internal functions, and freezing the variables at the best values, thereby producing an AI-based algorithm that can be downloaded to vehicles for traffic situation awareness in the field. Alternatively, the algorithm may be a different type of calculation tool, such as a computer program, an interpolatable tabulation of values, a graphical analysis tool, or other means for determining the coordinate listing from the images and the other input data.
At 552, a large number of traffic scenarios are imaged from multiple viewpoints, and the images and other data are fed as inputs to the AI structure. Variables in the AI structure may then be tuned or adjusted at 553 to obtain sufficiently accurate predictions of the traffic distribution under a wide range of conditions. Then at 554, when the AI model is sufficiently reliable for field applications, an algorithm can be derived from it and downloaded to vehicles, such as autonomous or semi-autonomous vehicles. The vehicles can then acquire and share images in traffic, apply those images to the algorithm, obtain a traffic map or coordinate listing from the algorithm, and thereby gain traffic situation awareness. The vehicles can then cooperate in avoiding collisions and regulating the flow of traffic. Optionally, the vehicles may record their image data and the predictions for further refinement of the model.
At 652, the first vehicle and the other vehicles acquire their images, each image showing various other vehicles and perhaps fixed items in proximity. The vehicles then transmit imaging messages to the first vehicle, including the images and the wireless addresses of the transmitting vehicles. At 653, the first vehicle performs viewpoint fusion by identifying objects that appear in multiple images, calculating the position of those objects by triangulation, and thereby combining the image data into a traffic map or coordinate listing of position coordinates. In addition, the traffic map or coordinate listing may include the wireless addresses of the vehicles that have supplied them. Further description such as vehicle type may also be provided. The first vehicle then broadcasts the traffic map as coordinate listing of the vehicles in proximate traffic, and optionally other entitles such as roadside wireless entities, in a mapping message. In addition, the first vehicle, or another entity, may generate a traffic map as an image, optionally annotated with the wireless addresses of the entities.
At 654, the vehicles have received the position data and wireless addresses of the other vehicles, as well as position data for the non-communicating vehicles. The vehicles can then communicate specifically with each other, to avoid accidents and manage the flow of traffic.
The wireless embodiments of this disclosure may be aptly suited for cloud backup protection, according to some embodiments. Furthermore, the cloud backup can be provided cyber-security, such as blockchain, to lock or protect data, thereby preventing malevolent actors from making changes. The cyber-security may thereby avoid changes that, in some applications, could result in hazards including lethal hazards, such as in applications related to traffic safety, electric grid management, law enforcement, or national security.
In some embodiments, non-transitory computer-readable media may include instructions that, when executed by a computing environment, cause a method to be performed, the method according to the principles disclosed herein. In some embodiments, the instructions (such as software or firmware) may be upgradable or updatable, to provide additional capabilities and/or to fix errors and/or to remove security vulnerabilities, among many other reasons for updating software. In some embodiments, the updates may be provided monthly, quarterly, annually, every 2 or 3 or 4 years, or upon other interval, or at the convenience of the owner, for example. In some embodiments, the updates (especially updates providing added capabilities) may be provided on a fee basis. The intent of the updates may be to cause the updated software to perform better than previously, and to thereby provide additional user satisfaction.
The system and method may be fully implemented in any number of computing devices. Typically, instructions are laid out on computer readable media, generally non-transitory, and these instructions are sufficient to allow a processor in the computing device to implement the method of the invention. The computer readable medium may be a hard drive or solid state storage having instructions that, when run, or sooner, are loaded into random access memory. Inputs to the application, e.g., from the plurality of users or from any one user, may be by any number of appropriate computer input devices. For example, users may employ vehicular controls, as well as a keyboard, mouse, touchscreen, joystick, trackpad, other pointing device, or any other such computer input device to input data relevant to the calculations. Data may also be input by way of one or more sensors on the vehicle, an inserted memory chip, hard drive, flash drives, flash memory, optical media, magnetic media, or any other type of file-storing medium. The outputs may be delivered to a user by way of signals transmitted to vehicle steering and throttle controls, a video graphics card or integrated graphics chipset coupled to a display that maybe seen by a user. Given this teaching, any number of other tangible outputs will also be understood to be contemplated by the invention. For example, outputs may be stored on a memory chip, hard drive, flash drives, flash memory, optical media, magnetic media, or any other type of output. It should also be noted that the invention may be implemented on any number of different types of computing devices, e.g., embedded systems and processors, personal computers, laptop computers, notebook computers, net book computers, handheld computers, personal digital assistants, mobile phones, smart phones, tablet computers, and also on devices specifically designed for these purpose. In one implementation, a user of a smart phone or WiFi-connected device downloads a copy of the application to their device from a server using a wireless Internet connection. An appropriate authentication procedure and secure transaction process may provide for payment to be made to the seller. The application may download over the mobile connection, or over the WiFi or other wireless network connection. The application may then be run by the user. Such a networked system may provide a suitable computing environment for an implementation in which a plurality of users provide separate inputs to the system and method. In the below system where vehicle controls are contemplated, the plural inputs may allow plural users to input relevant data at the same time.
It is to be understood that the foregoing description is not a definition of the invention but is a description of one or more preferred exemplary embodiments of the invention. The invention is not limited to the particular embodiments(s) disclosed herein, but rather is defined solely by the claims below. Furthermore, the statements contained in the foregoing description relate to particular embodiments and are not to be construed as limitations on the scope of the invention or on the definition of terms used in the claims, except where a term or phrase is expressly defined above. Various other embodiments and various changes and modifications to the disclosed embodiment(s) will become apparent to those skilled in the art. For example, the specific combination and order of steps is just one possibility, as the present method may include a combination of steps that has fewer, greater, or different steps than that shown here. All such other embodiments, changes, and modifications are intended to come within the scope of the appended claims.
As used in this specification and claims, the terms “for example”, “e.g.”, “for instance”, “such as”, and “like” and the terms “comprising”, “having”, “including”, and their other verb forms, when used in conjunction with a listing of one or more components or other items, are each to be construed as open-ended, meaning that the listing is not to be considered as excluding other additional components or items. Other terms are to be construed using their broadest reasonable meaning unless they are used in a context that requires a different interpretation.
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 63/260,814, entitled “Localization and Identification of Vehicles in Traffic by 5G Messaging”, filed Sep. 1, 2021, and U.S. Provisional Patent Application Ser. No. 63/243,437, entitled “V2X Messaging in 5G with Simultaneous GPS Reception”, filed Sep. 13, 2021, and U.S. Provisional Patent Application Ser. No. 63/245,227, entitled “V2X with 5G Image Exchange and AI-Based Viewpoint Fusion”, filed Sep. 17, 2021, and U.S. Provisional Patent Application Ser. No. 63/246,000, entitled “V2X Connectivity Matrix with 5G Sidelink”, filed Sep. 20, 2021, and U.S. Provisional Patent Application Ser. No. 63/256,042, entitled “Hailing Procedure for V2R, V2V and V2X Initial Contact in 5G”, filed Oct. 15, 2021, and U.S. Provisional Patent Application Ser. No. 63/271,335, entitled “Semaphore Messages for Rapid 5G and 6G Network Selection”, filed Oct. 25, 2021, and U.S. Provisional Patent Application Ser. No. 63/272,352, entitled “Sidelink V2V, V2X, and Low-Complexity IoT Communications in 5G and 6G”, filed Oct. 27, 2021, and U.S. Provisional Patent Application Ser. No. 63/287,428, entitled “V2X and Vehicle Localization by Local Map Exchange in 5G/6G”, filed Dec. 8, 2021, and U.S. Provisional Patent Application Ser. No. 63/288,237, entitled “V2X with 5G/6G Image Exchange and AI-Based Viewpoint Fusion”, filed Dec. 10, 2021, and U.S. Provisional Patent Application Ser. No. 63/288,807, entitled “V2X Messaging in 5G/6G with Simultaneous GPS Reception”, filed Dec. 13, 2021, and U.S. Provisional Patent Application Ser. No. 63/290,731, entitled “Vehicle Connectivity, V2X Communication, and 5G/6G Sidelink Messaging”, filed Dec. 17, 2021, all of which are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| 63260814 | Sep 2021 | US | |
| 63243437 | Sep 2021 | US | |
| 63245227 | Sep 2021 | US | |
| 63246000 | Sep 2021 | US | |
| 63256042 | Oct 2021 | US | |
| 63271335 | Oct 2021 | US | |
| 63272352 | Oct 2021 | US | |
| 63287428 | Dec 2021 | US | |
| 63288237 | Dec 2021 | US | |
| 63288807 | Dec 2021 | US | |
| 63290731 | Dec 2021 | US |
