Patent Application
20220020271
The document describes methods and systems that are directed to capturing images of a driving environment and using those images to help autonomous vehicles navigate in the environment.
Intersections are challenging for autonomous vehicles and human drivers alike. The traffic patterns are complex and there are occlusions from static structure (such as buildings, terrain, vegetation, or signs) and dynamic objects (such as cars, buses, or large trucks). While humans may rely on subtle clues in the environment (such as headlight reflections or shadows), or in some cases a specific piece of infrastructure to perceive activities around intersections despite some “blind spots” or hidden or occluded objects.
Therefore, for at least these reasons, a better method of navigational control of a vehicle through an intersection is needed.
In various embodiments, a node is provided for capturing information about moving objects at an intersection. The node may include a plurality of first cameras that are positioned to capture first digital images of an intersection from different fields of view within a first vision range. Each first camera captures a unique field of view having a field of view apex and a depth of field in a corresponding intersection direction. The node may include a second camera positioned to capture second digital images in a field of view that is wider than that of each first camera. The node may include a processor and a computer-readable storage medium including programming instructions that are configured to, when executed, cause the processor to: i) detect in the first digital images and the second digital images a set of objects of interest of the intersection; ii) determine motion of each detected object of interest in the set of objects of interest from consecutive images of the first digital images or the second digital images separated in time; and iii) generate, for each object of interest of the set of objects of interest, augmented perception data that includes location data in the global coordinate system and the determined motion of each object of interest in the set of objects of interest. The node may include a communication system configured to transmit, via a wireless communication system, the augmented perception data to a remote server system. The augmented perception data is accessible by at least one vehicle to control navigation of the at least one vehicle through the intersection.
In some embodiments, the plurality of first cameras may include a plurality of narrow field of view cameras, each of which is positioned to capture digital images of a unique segment of the intersection. The second camera may include a wide field of view camera that is positioned to capture digital images that include substantially all of the unique segments of the intersection.
In some embodiments, the wide field of view camera may include a convex lens.
In some embodiments, the second digital images may include a hemispherical image.
In some embodiments, the second digital images may include a panoramic image.
In some embodiments, the computer-readable medium may include additional programming instructions that are configured to, when executed, further cause the processor to: iv) segment each of the second digital images into a plurality of image segments; and v) detect, in at least one of the plurality of image segments, an object of interest in the second vision range. The set of objects of interest of the intersection includes the object of interest in the second vision range.
In some embodiments, the second camera is further positioned to capture the second digital images in a volume of space at the intersection vertically below the field of view apex of each first camera. The second camera may include a second vision range of 360°.
In some embodiments, the computer-readable medium may include additional programming instructions that are configured to, when executed, cause the processor to: iv) process the first digital images of the first vision range to extract at least one object of interest in the first vision range; and v) process the second digital images of the second vision range to extract at least one object of interest in the second vision range. The second vision range is different from the first vision range. The set of objects of interest of the intersection may include the at least one object of interest extracted from the first vision range and the at least one object of interest extracted from the second vision range.
In some embodiments, the node may further include a node housing configured to house the plurality of first cameras and the second camera. The node may further include a node controller housing configured to house the processor, the computer-readable storage medium and at least a portion of the communication system. The node may further include a node mount assembly configured to attach the node housing and the node controller housing to a gantry of a traffic light pole.
In some embodiments, the communication system transmits the generated augmented perception data to a first server of the remote server system. The communication system may transmit at least one of the first digital images and the second digital images to a second server of the remote server system.
In some embodiments, the communication system may communicate with one or more nodes of a network of nodes.
In some embodiments, the computer-readable medium may include additional programming instructions that are configured to, when executed, cause the processor to: vi) detect a situational flag condition associated with at least one object of interest in the set of objects of interest proximate the intersection; and v) flag the generated augmented perception data associated with the at least one object having the detected situational flag condition. The communication system may transmit the flagged augmented perception data to at least one of an adjacent node of the network of nodes, a local law enforcement computing system, a local healthcare services computing system or a first responder computing system.
In some embodiments, the computer-readable medium may include additional programming instructions that are configured to, when executed, further cause the processor to: iv) classify each object of interest of the detected set of objects of interest as one of a moving actor, a moving object or a moving vehicle. The instructions that cause the processor to i) determine the motion of each object of interest in the set of objects of interest, includes instructions that are configured to, when executed, further cause the processor to: a) forecast a direction and a speed of motion of each object of interest of the set of objects of interest. The flagged augmented perception data may include the forecasted direction and speed of motion associated with the at least one object having the detected situational flag condition.
In some embodiments, the computer-readable medium may include additional programming instructions that are configured to, when executed, further cause the processor to: iv) classify each object of interest of the detected set of objects of interest as one of a moving actor, a moving object or a moving vehicle. The instructions that cause the processor to ii) determine the motion of each object of interest in the set of objects of interest, includes instruction that are configured to, when executed, further cause the processor to: a) forecast a direction and a speed of motion of each object of interest of the set of objects of interest. The augmented perception data, for each object of interest of the detected set of objects of interest, includes the forecasted direction and speed of motion.
In some embodiments, the computer-readable medium may include additional programming instructions that are configured to, when executed, further cause the processor to: iv) determine a location of each object of interest in a corresponding image of the first digital images and the second digital images; and v) translate the determined location into location data in a global coordinate system.
In this document, when terms such as “first” and “second” are used to modify a noun, such use is simply intended to distinguish one item from another, and is not intended to require a sequential order unless specifically stated. The term “approximately,” when used in connection with a numeric value, is intended to include values that are close to, but not exactly, the number. For example, in some embodiments, the term “approximately” may include values that are within +/−10 percent of the value.
In this document: (i) the term “comprising” means “including, but not limited to”; the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise; and (iii) unless defined otherwise, all technical and scientific terms used in this document have the same meanings as commonly understood by one of ordinary skill in the art. Also, terms such as “top” and “bottom”, “above” and “below”, and other terms describing position are intended to have their relative meanings rather than their absolute meanings with respect to ground. For example, one structure may be “above” a second structure if the two structures are side by side and the first structure appears to cover the second structure from the point of view of a viewer (i.e., the viewer could be closer to the first structure).
Many current commercial solutions for traffic detections use extremely naive approaches. When they are camera based, they are based on traditional image processing (background subtraction, lane occupancy), or use different sensing modalities (ground loops, radar).
While the current systems are extremely expensive and costly to deploy and maintain, the smart node system described in this document and its associated sensing may be by design low-cost to enable a rapid and cost-effective deployment. This is achieved by improving the computing architecture and relying on a cost-effective sensor which this document may refer to as a “smart node.” Each node is configured to perceive activities around an intersection and provide an autonomous vehicle information which may be in “blind spots” of, or hidden from the vehicle's computer vision system.
Before describing the system in detail, it is helpful to establish a few acronyms:
APD=augmented perception data;
AV=autonomous vehicle;
AVDS=autonomous vehicle driving system;
CVS=computer vision system;
FOV=field of view;
GLS=geographic location system;
GPS=Global Positioning System;
GPU—graphics processing unit;
OOI=object of interest;
TL=traffic light;
VR=vision range.
These terms will also be defined when first used below.
Referring now to
In some embodiments, the smart node 170 comprises a combination of cameras and computing hardware which may be situated near an intersection (e.g., mounted on traffic light gantries). The smart node 170 may be configured to detect, classify and track multiple actors and objects (including, for example: vehicles, pedestrians, and cyclists) in its fields of view (FOVs). The smart node 170 may forecast a moving actor or moving object's motion, based on real time images captured in the FOVs and communicate the forecast to at least one vehicle 105 in-range and/or passing through the intersection. A smart node 170 can be interconnected to provide local APD 175 which may also be used with city-wide information pertinent to routing and planning of the vehicle 105. The terms “APD” and “APD information” may be used interchangeably herein.
The vehicle 105 may be an AV as shown driving along the road 110. In some embodiments, the vehicle 105 may be a semi-autonomous vehicle. As the vehicle 105 drives, a vehicle CVS 115 incorporated into the vehicle 105 is configured to receive a digital image of at least one traffic signal device 130 and other objects in the environment while driving along a path or route and passing through an intersection. The vehicle CVS 115 may include one or more cameras for capturing digital images of various features of the environment in which the vehicle 105 is traveling, along with a processor and software for processing images and identifying objects of interest in the images, to define a vehicle's VR.
The term “route” as used herein means the overall or end-to-end path from point A (point of origination) to point B (point of destination) that the vehicle will travel to arrive at the point of destination. The term “path” means the immediate or imminent path the vehicle intends to take over the next L meters along the “route,” where L is a non-zero number. The term “path” as used herein represents the un-traveled portion of the route and will sometimes be referred to as an “imminent path.” Once navigation of the vehicle begins to proceed along the route a particular distance, that traveled distance of the route may sometimes be referred to as a “traveled portion.”
Each camera includes a FOV, only one field of view is shown and denoted by dashed lines 127 and 127′ for the sake of illustration. Such captured features may include one or more traffic signal devices 130 and at least one of OOI (i.e., moving actors, moving objects, moving vehicles, stationary objects, and stationary vehicles) in the environment. However, as the vehicle 105 moves within an environment, each moving actor, moving object and moving vehicle needs to be registered, classified and tracked to determine their motion, such as location, direction and speed of moving. The APD may include the classification of the OOI, which may be used locally by the vehicle for local vehicle navigation and control. However, the classification of the OOI may be used by a remote server system 155 for use in city-wide motion planning, as will be described in more detail herein. The remote server system 155 may include one or more servers and memory.
The system 100 may include the vehicle CVS 115 incorporated into the vehicle 105. The road 110 and lane traffic light control 111 may be separate from the system 100 and part of the environment. The lane traffic light control 111 may include one or more traffic signal devices 130. The lane traffic light control 111 is denoted in a dashed box. The system 100 may include a GLS 160 configured to determine a location and orientation of the vehicle 105. The GLS 160 may include a GPS device. The GLS 160 may be implemented using hardware, firmware, software or a combination of any of these. For instance, GLS 160 may be implemented as part of a microcontroller and/or processor with a register and/or data store for storing data and programming instructions, which when executed, determines a location and orientation of the vehicle. It is noted, however, that other forms of geographic location determination systems may additionally, or alternatively, be used. The GLS 160 may be incorporated into the vehicle 105. A respective one smart node 170 is shown located at or near the lane traffic control 111, as in a scenario the smart node 170 may be mounted on the traffic light gantry radiating from a pole at an intersection.
The system 100 may further include a transceiver 120 incorporated in the vehicle 105. The transceiver 120 may include a transmitter and receiver configured to send and receive digital information from a remote server system 155 via a wired and/or wireless connection such as, for example, through the Internet 150, where the vehicle 105 and the remote server system 155 are in electronic communication with each other. The remote server system 155 may be part of a cloud computing system. The system 100 may further include a processor 125. The processor 125 may be configured to represent the traffic signal device 130 and other objects as a raster image. It is noted that the processor 125 may be a standalone processor, the vehicle's processor, and/or the remote server's processor. Data processed by the processor 125 may be data received from the vehicle 105, received from the remote server system 155, received from any number of smart nodes 170 and/or a combination of data from the vehicle 105, the smart nodes 170 and the remote server system 155. However, for the sake of illustration, the processor 125 is represented incorporated in the vehicle 105. The vehicle 105 may include a standalone processor (e.g., processor 125) and/or at least one separate vehicle processor.
According to various embodiments, the system 100 may include the vehicle 105. The vehicle 105 may include an autonomous vehicle driving system (AVDS) 265 for a fully autonomous vehicle or semi-autonomous vehicle with a computer-assisting driving system to assist a human operator of the vehicle. The AVDS 265 may be implemented using hardware, firmware, software or a combination of any of these. For instance, AVDS 265 may be implemented as part of a microcontroller and/or processor with a register and/or data store for storing data and programming instructions, for autonomous vehicle driving, route navigation and collision avoidance and/or the like.
The AVDS 265 may control a braking system (not shown), engine system (not shown), and/or steering system (not shown) of the vehicle 105 in response to at least one control signal representative of the classification state of the current instantiation of a traffic signal device 130, for example, as will be described in more detail in relation to
The AVDS 265 may include a system architecture 1000 as will be described in more detail in relation to
The system 100 may be configured to provide descriptions of the different actors in an intersection, with different classes of objects, location, heading, velocity and other relevant information that may be used to help guide a vehicle 105 which is at or near or approaching the intersection monitored by the smart node 170. For a more precise integration with the AVDS of the vehicle 105, the smart node 170 may express the collected data in a reference frame common to the AVDS of the vehicle 105 and the node 170, likely through the use of high definition maps.
As will be seen from the description herein, the vehicle CVS of the vehicle, receives at least one digital image of an environment along a planned route. The vehicle CVS has a vehicle's VR which is generally updated as the vehicle moves. A processor of the vehicle detects, in the at least one digital image, a first set of OOIs and determines motion of each OOI in the first set of OOIs. The vehicle or processor receives APD associated with an in-range node to and along a portion (i.e., the imminent path) of the planned route, the in-range node has a node CVS. The received APD identifies motion of each OOI of a second set of OOIs detected within a node's VR. The vehicle's VR and the node's VR are different vision ranges. The processor of the vehicle controls motion of the vehicle to and along the portion (i.e., the imminent path) of the planned route based on a fusion of the first set of OOIs and the second set of OOIs.
In some instances, the wide FOV camera 210 may include a convex lens 212. The lens may be configured to create a hemispherical image. The convex lens 212 may include a fisheye lens or a hemispherical lens. In other embodiments, the lens 212 may include multiple lenses configured to create a panoramic image in 360°, for example, from multiple wide angle images that can be stitched together. The convex lens 212 may comprise a principle axis 445 (
The multiple FOVs will be described in more detail in relation to
The smart node 170 may collect information to understand traffic at intersections, even when no vehicles 105 are nearby, to improve routing or decision making, such as by the remote server system 155, the vehicle 105 or fleet controller.
The FOV pattern 300 of the node CVS 205 may include a first FOV defined by the angle between lines 322 and 322′ and in the direction of the arrows of the lines 322 and 322′. The FOV pattern 300 of the node CVS 205 may include a second FOV defined by the angle between lines 324 and 324′ and in the direction of the arrows of the lines 324 and 324′. The FOV pattern 300 of the node CVS 205 may include a third FOV defined by the angle between lines 326 and 326′ and in the direction of the arrows of the lines 326 and 326′. The FOV pattern 300 of the node CVS 205 may include a fourth FOV defined by the angle between lines 328 and 328′ and in the direction of the arrows of the lines 328 and 328′. In some scenarios, the angle of any one of the first, second, third and fourth FOVs may vary such that the any one FOV may be blocked by buildings, trees or structures along the at least one lane 310 and at least one lane 313. Likewise, the angle of any one of the first, second, third and fourth FOVs may vary such that the any one FOV may be enlarged by the absence of buildings, trees or structures along the at least one lane 310 and at least one lane 313.
In
The definition of a camera's VR for camera 315 is defined as a FOV and angle of view associated with the camera's image capture sensor and lens configurations. The vehicles CVS 115 (
Returning again to
In many locations, each intersection does not include a light. Therefore, the first, second, third and fourth FOVs by the intersection direction cameras may extend through adjacent intersections along the at least one lane 310 and/or 313 until the next smart node along the at least one lane 310 or 313. For example, in the illustration, lane 309 intersects each of the lanes 312, 313 and 314. Lane 310 intersects each of the lanes 312, 313 and 314. Moreover, lane 311 intersects each of lanes 312, 313 and 314. The intersection between lane 310 and lane 313 also has a smart node 170.
The node (i.e., node 170) is configured to capture segmented perception data at an intersection. The node comprises a plurality of first cameras (i.e., cameras 207) that are each positioned to capture first digital images of an intersection from different FOVs within a first vision range. With reference to the first vision range of the node, the horizontal FOV would include the FOV including the unique segment fields of view denoted as FOV1, FOV2, FOV3 and FOVX. In some scenarios, the unique segment fields of view denoted as FOV1, FOV2, FOV3 and FOVX may have portions of the fields which overlap. Each first camera (i.e., camera 207) captures a different field of view having at a field of view apex (i.e., origin 429) and a depth of field to a different intersection direction, as shown in
The node (i.e., node 170) may be configured to detect, in at least one the first digital images and the second digital images, a set of objects of interest surrounding the intersection. The node (i.e., node 170) may be configured to determine motion of each object of interest in the second set of objects of interest and generate APD for each object of interest of the second set of objects of interest. In the illustration of
The node (i.e., node 170) may comprise a communication system (i.e., modem 245 and communication unit 636) configured to communicate wireless communications within a communication range around the intersection including to transmit the APD to vehicles within the communication range of the communication system. The APD may include the determined motion and location data in a global coordinate system of each object of interest in the second set of objects of interest.
The communication system may be further configured to communicate with one or more nodes of a network of nodes (
In the illustration, a moving actor MA41 is capable of being imaged by the wide FOV camera 210 but not the intersection direction camera 207X or any other intersection direction camera until the moving actor MA41 comes into view of the lens of the intersection direction camera 207X. Accordingly, there is a volume of space directly below and surrounding the smart node 170 which may not be able to capture images of moving objects or OOIs by the intersection direction camera 2073, 2072, 2073, . . . , 207X. Thus, any moving object may appear in a field of view of one of the intersection direction cameras and disappear. However, the wide FOV camera 210 is configured to capture images within the hidden volume of space and at least an overlapping portion of the narrow FOV of each intersection direction camera of the intersection direction cameras 207 to that motion tracking and forecasting of one or more OOIs is not interrupted when hidden from the intersection direction cameras 207.
Regarding
Referring again to
The communication device 245 may include a LTE modem or other communication device configured to communicate using a wireless communication protocol, which may be part of a communication system. The communication device 245 may communicate the APD 175 to the vehicle 105. The APD 175 may include one or more packets with one or more APD fields. For example, each APD field may include information associated with a different one OOI in the environment in-range of the smart node 170. An OOI may include stationary objects, moving objects, moving actors, moving vehicles. The “in-range” may be based on the distance or range the intersection direction cameras can capture images. For example, if the intersection direction camera can capture images at a distance of up to 20 meters from the camera, the vehicle may be in-range when the vehicle is 20 meters from the intersection or node location. The node controller 225 may include an edge machine learning (ML) accelerator interfaced with the cloud computing system or remote server system 155 via the Internet 150. The smart node 170 may include a GPS 230 or have fixed location coordinates stored in memory of the local storage device 240.
In this illustration, the environment includes moving actors MA51, MA52, MA53 and MA54. The environment includes stationary objects SO51, SO52, SO53, SO54 and SO55. The stationary objects SO51 and SO52 include buildings, for example. Stationary objects SO53, SO54 and SO55 may be trees. In this illustration, the stationary objects SO51, SO52, SO53, SO54 and SO55 are on either the right or the left side of road 510 of the path driven by the vehicle 105.
In the illustration, the intersection environment includes moving objects MV51 and MV52. As an example, the moving object MV51 is obscured by stationary object SO51. Therefore, the vehicle CVS 115 of the vehicle 105 is not capable of imaging the obscured stationary object SO51. However, the obscured stationary object SO51 relative to the vehicle CVS of the vehicle 105 is within the field of view of at least one intersection direction camera 207 and/or the field of view of the wide FOV camera 210 of the node CVS 205 of the node 170.
The moving actors MA51, MA52, MA53 and MA54, the stationary objects SO51, SO52, SO53, SO54 and SO55, and the moving objects MV51 and MV52 are examples of OOIs in the environment and in-range of the node 170.
In operation, the smart node 170 may determine and track at least one of the location, direction and speed of the moving vehicle MA51 hidden behind a building (i.e., stationary object SO51) relative to vehicle 105. In operation, the smart node 170 may determine and track at least one of the location, direction and speed of a moving bicycle (i.e., moving vehicle MA52) hidden by trees (i.e., stationary objects SO53, SO54 and SO55). The vehicle 105 may automatically update its on-board computing device 1010 (
In this illustration, moving actors MA51, MA52, MA53 and MA54 are within the view of the vehicle CVS 115 of the vehicle 105 and the node CVS 205. However, if moving actors MA54 is below the node CVS 205 of the node 170, the moving actors MA54 may be out of view from all the intersection direction cameras but in the field of view of the wide FOV camera. However, if the moving actors MA54 moves behind the stationary object SO51, the moving actors MA54 may be out of view of the vehicle CVS 115 of the approaching vehicle 105 but in view of one or more intersection direction cameras of the node 170. In the illustration, moving actors MA54 include an adult and child together. Therefore, the motion and speed estimation may be based on the speed of a child and not the adult.
The APD 175 may be communicated to the remote server system 155 and received in advance of the arrival of a vehicle 105 at the intersection, in response to a query for current APD information associated with the intersection. The current APD information provides the on-board computing device 1010 (
In the environment, a moving actor may be tracked to a stationary vehicle (i.e., parked car) in the environment. Assume that moving vehicle MV51 is stationary. However, as the once moving actor opens a vehicle door, such moving vehicle door becomes a moving object which can be reported to an approaching vehicle in-range of the smart node 170, especially if the moving vehicle door is opening up into the imminent path to be driven by the vehicle. The APD 175 may be representative of information associated with the moving object (i.e., moving car door). The APD 175 relative to the moving car door may be important to the on-board computing device of the vehicle depending on the location of the moving object relative to at least one of the speed and location of the approaching vehicle. Some vehicle doors may open into the imminent path of an approaching vehicle. Then, once the vehicle door closes, the stationary vehicle may be denoted as a pending moving vehicle. The node CVS 205 may use image tracking and machine learning to predict, in some embodiments, when a parked vehicle may become a moving vehicle based on timing a moving actor moved into a driver's seat of the parked vehicle. In such a scenario, lights of the parked vehicle may turn on or flash as the vehicle turns on which may be captured by the node CVS 205. Once the vehicle becomes a moving vehicle, the APD 175 subsequently reported or updated may be representative of information associated with the moving vehicle, including location, direction and speed. The APD 175 for each OOI and especially those OOI hidden from any one of the vehicles in range of the node 170 can be used by the AVDS 265 to refine its navigational guidance based on off-board advanced notification of obscured OOI, such as an obscured OOI with a forecasted motion toward or in the imminent path of the vehicle 105, by way of non-limiting example. While, a vehicle is in-range of the smart node 170, the APD 175 of an OOI may be representative of (i) at least one of an obscured OOI relative to vehicle CVS 115 of the vehicle, (ii) out-of-FOV range OOI of the vehicle CVS 115 of the vehicle relative to the imminent path to be driven by the vehicle and/or (iii) vehicle identified OOIs discovered, classified, and tracked by the AVDS. An obscured OOI may be obscure from the vehicle CVS 115 and is defined as an OOI which is hidden from view by objects or structures within the environment, for example. An OOI that is out-of-FOV range of the vehicle CVS 115 represents an OOI that is capable of being imaged using the cameras of the node CVS of the intersection but out of the FOV range of the cameras of the vehicle CVS 115.
Thus, the system 100 uses advanced processing capabilities at the smart node 170 instead of relying on off-board capabilities. The smart node 170 may only require a single sensing modality (i.e. cameras) to provide perception augmentation directly or indirectly to in-range vehicles 105. The perception augmentation is provided indirectly to the in-range vehicles 105 via the remote server system 155.
The node 170 may include a node CVS 205 comprising narrow FOV cameras 207 and a wide FOV camera 210 which produce raw images. The raw images may be sent to the node controller 225 which may include a processor 230 (
The one or more processing channels 620 may process the raw images from the node CVS 205 using signal processing algorithms 630 executed by the processor (sometimes referenced in the figures as “ISP 630”). The signal processing algorithms 630 may include, without limitation, debayer filtering algorithms 634 such that raw images become red, green, blue (RGB) images. In one or more embodiments, the signal processing algorithms 630 may also employ tonal mapping algorithms 632 to approximate an appearance of images such as for generating processed image data representative of high dynamic range images. The node controller 225 may process the raw images from each of the cameras 207 and 210 in parallel by the processing channels 620 such that each channel processes a different raw image stream of a respective different camera. One of the processing channels 620 may include a segmentation module (SM) 625, denoted in a dash, dot, dot box, configured to segment the omni-directional image (i.e., image 710) captured by the wide FOV camera 210. The dash, dot, dot box of the segmentation module 625 denotes that it may not be used in other channels. The dash, dot, dash line from cameras 207 denotes that the camera output in some channels may be fed into the signal processing algorithms 630 of those channels.
The smart node 170 may include an encoder 635. For example, the encoder 635 may be an H-265 encoder configured to encode data using an H.265 encoder protocol. The image data output by execution of the signal processing algorithms 630 may be sent to the encoder 635. The encoder 635 may be implemented using hardware, firmware, software or a combination of any of these. For instance, encoder 635 may be implemented as part of a microcontroller and/or processor with a register and/or data store for storing data and programming instructions, which when executed, performs the encoding of the image data using an encoding protocol.
The image data output from the encoder 635 may be in the format representative of a video stream or still images. The output of the encoder 635 may be sent, via a communication unit 636, to a traffic control server 670 of the remote server system 155, for example, and archived in memory 673. The communication unit 636 may include a modem, transmitter, receiver and/or network card to communicate with the remote server system 155 using a wireless or wired communication protocol via the Internet, for example.
The processing channel(s) 620 may process the image data using a deep machine learning (DML) network 637. The deep machine learning network 637 may be implemented using hardware, firmware, software or a combination of any of these. For instance, the deep machine learning network 637 may be implemented as part of a microcontroller and/or processor with a register and/or data store for storing data and programming instructions, which when executed, performs the deep machine learning or performs machine learning algorithms for identifying and classifying OOIs. The deep machine learning network 637 may be configured to perform feature extraction associated with situational awareness of currently moving objects, moving vehicles, moving actors, as well as, stationary objects, stationary vehicles and stationary objects which have a propensity for motion. The deep machine learning network 637 may include an OOI classifier 638. The feature extraction may find, identify, and locate OOIs in each captured image using one or more feature extraction algorithms. The term “locate” as used herein may relate to geographical coordinates to locate the OOI in a global coordinate system.
Feature extraction algorithms may include, without limitation, edge detection, corner detection, template matching, dynamic texture processing, segmentation image processing, motion detection, frame difference methods, object tracking, background subtraction, object recognition and classification, etc. By way of non-limiting example, the background subtraction algorithm may include an adaptive Gaussian mixture model. Motion tracking algorithms may determine a multiple object tracking precision (MOTP) metric and a multiple object tracking accuracy (MOTA) metric to improve performance of the motion tracking by the node or vehicle. The object tracking and motion tracking may use correlation filters and affinity matrix methodologies, by way of non-limiting examples.
By way of non-limiting example, when an OOI is located in an image, the location data may be in GIS spatial raster format which may subsequently be translated to a GIS spatial vector format. When detecting motion of an OOI, captured images with the OOI separated in time may be analyzed for motion. The captured images may be of the same intersection direction or region of interest within a field of view of a camera. In the embodiments, the node cameras are generally stationary. An object may be identified in a first image, such as in a frame of the first image. The object may be tracked in the next subsequent or consecutive image separated in time from the first image. For example, pixel-based frame differencing, background subtraction, etc. may be used to detect motion of an OOI in an intersection direction toward or away from the intersection. Each OOI of a set of OOIs may be represented as a motion vector corresponding to the motion direction and a velocity or speed magnitude.
The OOI may be represented in GIS spatial raster data. The OOI GIS spatial raster data may be translated to GIS spatial vector data with longitude and latitude coordinates for example. The process for translating or converting GIS spatial raster data to and from GIS spatial vector map data is known in the art. The difference between the OOI GIS spatial raster data or OOI GIS spatial vector map data of the separated in time images may provide a distance traveled value, such as in miles. The distance traveled value may be a function of a reference point of the node at an intersection or some other reference point associated with the intersection. Furthermore, the interval of time (e.g., fraction of hour, minutes or seconds) between the time of the capture of the first image and the time of the capture of the next subsequent image may be used to determine the speed or velocity of the OOI under evaluation to determine the miles per hour (MPH) or some other metric denoted speed or velocity.
The OOI location includes global coordinates of a global coordinate system. The vehicle may determine a first set of OOIs based on the vehicle's VR. The node may determine a second set of OOI based on the node's VR. By way of non-limiting example, at least one OOI in the second set of OOIs may include global location coordinates which are outside of the vehicle's VR immediately before or at the time the APD is received by the vehicle from the remote server system 155. In some scenarios, the APD is received by the vehicle from an in-range node. The controlling of the motion of the vehicle may include using the APD of the at least one OOI in the second set of OOIs having the location coordinates outside of the vehicle's VR.
The deep machine learning network 637 may include an intersection situational data flag generator (ISDF GEN) 639. The flag generator 639 may determine which OOI classification from the OOI classifier 638 should be flagged as intersection situational data so that such information may be sent to other remote computing systems via an intersection situational data flag packet, as will be discussed in relation to
The processing channel(s) 620 may include an image tracker 643 which receives the information associated with an extracted one OOI and to create the APD 175 which is time stamped with a global time reference via the global time module 640. Thus, the image tracker may be configured to determine a speed for each moving OOI. The GPS 250 may provide GPS signals to a global time module 640 where the local time is synchronized with the GPS timing data. The image tracker 643 may track OOIs associated with the FOV within the multiple FOVs for the generation of the respective APD for each corresponding OOI processed.
Each processing channel 620 may include a map projection module 645. The smart node 170 may use high definition maps. The node 170 may store vector map data in database 647 and calibration data in database 649 where the databases include a memory storage device. The vector map data may include vector-based geographic information system (GIS) data about the earth. The vector map data may include coordinate reference system of global coordinates associated with a longitude and latitude of the earth. The vector map data may include major road networks, geographic boundaries, elevation data, etc. The vector map data may be updated dynamically. The map projection module 645 may include a periodic map registration process as changes to the environment occur. The map projection module 645 may convert or translate location data of an OOI from two-dimensional (2D) image space (raster image coordinates) to a three-dimensional (3D) world or map space of a global coordinate system. The map projection module 645 may receive GIS spatial data in a raster image format associated with the image. The image data of an OOI is mapped (i.e., converted or translated) from a 2D image location (i.e., pixel coordinates x, y) (GIS raster spatial data) to a 3D location with latitude, longitude and altitude coordinates (GIS vector spatial data) of a global coordinate system. The smart node 170 may further include local storage 240 as previously described in relation to
In at least one scenario, the smart node 170 may include detection and tracking software based on machine learning algorithms to provide an understanding (situational awareness) of the environment around and beyond an intersection using a single sensing modality via the node CVS 205 by tracking at least moving actors and moving objects in the environment and by determining their location, direction of motion and speed of travel.
Referring now to
The tracker fusion management server 655 may provide augmented local tracks information (i.e., APD information) to the vehicle 105, in response to receiving a query from the vehicle 105. The tracker fusion management server 655 acts as a broker between those smart nodes 170 associated with the fusion management server 655 and those vehicles 105 traveling to and along an imminent path including the smart nodes at certain intersections. In some embodiments, instead of having each smart node 170 communicate the APD to the in-range vehicles 105 at any instantiation, the smart node 170 may communicate the APD to the tracker fusion management server 655 in order to reduce the amount of the communications and communication data costs at the node. The tracker fusion management server 655 may then communicate the received APD information to vehicles 105 that are in communication with the tracker fusion management server 655. The received APD may be sent to the vehicle, in response to a specific query initiated by the vehicle for the real-time APD of a particular traffic light the vehicle is approaching.
With reference to
The networked track client 680 connects to and queries the tracker fusion management server 655. The network track client 680 receives augmented local tracks information, in response to the vehicle's query, from the tracker fusion management server 655. The received node's APD may be combined with the vehicle's own perception data by the network track client 680 and feed into the motion planning engine 686 and smart routing engine 682 of the autonomous vehicle navigation controller 1020 (
The tracker fusion management server 655 may receive tracks on a map (ToM) with velocity from at least one smart node 170. The tracks on the map information includes the APD information converted into the 3D global coordinates. The tracker fusion management server 655 may fuse the augmented perception data from adjacent nodes of the network of nodes in response to a search, each adjacent node may have a respective different node vision range associated with coordinates of the imminent path.
The smart routing engine 682 may be configured to receive data representative of traffic conditions (TC) and TL states related to routing for dynamic adjustment to a planned route using real time information from the smart node network 800 (
The gateway 660 may be a broker agent or server to aggregate received information about traffic light signal devices such as without limitation, dynamic timing control and/or traffic light signal control schedules. For example, traffic light signal devices may have a different timing sequence during rush hour time intervals than at other times. Information from the traffic control infrastructure 679 may come from different sources. Traffic light signal timing may be used by the machine learning modules for the traffic light classification process or the classification training process. The traffic light timing sequence may change from one city, town or state to another. Thus, the gateway 660 may sometimes be referred to as “gateway server 660.”
The tracker fusion management server 655 may communicate with the traffic control server 670 configured to receive the video stream or digital images from the nodes 170 in the network 800 (
In some scenarios, the vehicle may receive traffic conditions and TL states from the gateway server 660. The traffic condition information may be provided from the traffic control server 670 based on assembled data from the smart nodes 170 directly and indirectly via the tracker management server 655. The vehicle 105 may provide current vehicle position information and route information to the gateway server 660 so that the TL states are for those traffic lights that are along the imminent path of the remaining portion of the planned route yet to be traveled.
The transceiver 120 of the vehicle 105 is configured to receive, from a remote server (i.e., traffic control server 670) network traffic information representative of traffic condition detected within the VR of any node of the nodes. The on-board computing device 1010, via the navigation controller 1020 (
The smart node's APD information to the remote processing system 650 may be fused by the tracker fusion management server 655 where the tracker fusion management server 655 and/or traffic control server 670 may then detect such traffic congestion or traffic accident. For example, a smart node 170 providing information about three or four OOIs of the moving vehicle type in the narrow FOV of the intersection direction cameras may be representative of low traffic congestion conditions at the intersection of the reporting smart node 170. However, information about 30-40 OOIs of the moving vehicle type may be representative of high traffic congestion conditions at the intersection of the reporting smart node 170. As the number of OOIs of the moving vehicle type increases at adjacent smart nodes, the tracker fusion management server 655 may fuse such data to forecast traffic conditions which can be downloaded to the on-board computing device 1010 (
In a scenario, adjacent nodes 170 may communicate with each other to track a vehicle of interest which may be involved in an accident but left the scene. In other words, each respective node 170 may be configured to further classify a moving vehicle as a flagged object of interest. The APD information of a flagged object of interest may be flagged as intersection situational data for additional tracking by adjacent nodes 170. Furthermore, image information associated with the flagged object of interest may be flagged as intersection situational data for law enforcement purposes and sent to a local law enforcement computing system 810 to report an accident. The OOI classifier 638 may generate the flag based on the type of intersection situational data identified.
In a scenario, as the each node 170 detects and tracks OOIs, an OOI may be further classified as in need of medical attention, the node 170 may flag the image information and/or APD information of the respective OOI as intersection situational data for communication to a local healthcare service computing system 820, local law enforcement computing system 810 and/or first responder computing system 830. For example, in the event of a robbery, the node 170 may flag the image information and the APD information for all OOIs involved for reporting to the local healthcare service computing system and the local law enforcement computing system, for example. The term all OOIs, in this instance, may include witnesses, victim and assailant, by way of non-limiting example. A first responder computing system 830, such as that of a local fire department or other first responder agency, may be in communication with each node 170.
In a scenario, an OOI may include a weapon where the node 170 may be configured to further classify the weapon type of the OOI which in some cases may be flagged as intersection situational data. In a scenario, multiple OOIs may be grouped into a single OOI, such that the single OOI of a group is tracked collectively. This has applications for a group of bicyclist passing through an intersection or a group of runners. Specifically, the motion of the single OOI based on a group of OOIs may be predicted or forecasted based on a classified group type.
The local law enforcement computing system 810, the local healthcare services computing system 820 and/or first responder computing system 830 may be web-based systems configured to receive internet communications via the Internet 805 or World Wide Web (WWW) from the network 800 or each node 170, for example. The local healthcare services computing system 820 may include an ambulance service. The local law enforcement computing system 810 may include communications to the local police, local sheriff, and/or state troopers. The local law enforcement computing system 810, the local healthcare services computing system 820 and/or first responder computing system 830 may receive flagged information as generated by the intersection situational data flag generator 639 (
The method 900 may include, at block 902, capturing images of a monitored intersection by a smart node 170. The monitoring includes capturing images from multiple FOVs. The images may be captured by the cameras of the node CVS 205 (
The method 900 may include, at block 906, classifying each OOI. The classification may be performed by the deep machine learning network 637, as well. The classifying may include a support vector machine (SVM) mode, or deep neural networks or other computer vision tool. For example, an OOI may be a moving vehicle, a moving object, a stationary vehicle, stationary object, moving actors and stationary actors at any instantiation of monitoring. Classifying different types of OOIs allows various aspects of motion for each OOI to be determined, such as, location, heading, velocity and other relevant information for each classified object of interest. In some scenarios, the deep machine learning network 637 may flag OOIs based on the classification of the OOI.
The method 900 may include, at block 908, for each OOI, forecasting, by the smart node 170, the classified OOI's motion, the motion of an OOI may include a direction of motion. The method 900 may include, at block 909, for each OOI, converting the 2D location data from the image to 3D global coordinates (i.e., latitude, longitude and altitude coordinates) using the vector map data 647. The method 900 may include, at block 910, for each OOI, generating the APD 175 of each OOI. The information associated with the APD 175 of any one OOI in the environment may include the OOI classification, the OOI forecasted motion (including direction), OOI current speed, and/or OOI location data. The method 900 may include, at block 912, communicating, by the smart node 170, the APD 175 associated with each OOI in the intersection and in-range of the smart node 170 to the tracker fusion management server 655. For example, modem 245 (
In some instances, the assembled packet may be sent to multiple services. Furthermore, the local law enforcement computing system may include a computing system for a local law enforcement agency, a state law enforcement agency, and/or federal law enforcement agency. For example, some roads or highways are state controlled while other roads are federally controlled, each with a different responsible law enforcement agency.
The method 930 may include, at block 944, determining whether to notify adjacent nodes of a flagged OOI. If the determination is “NO,” the method 930 may end at block 950. If the determination at block 944 is “YES,” the method 930 may include, at block 946, identifying the next smart nodes in the predicted path of the flagged OOI or identify adjacent nodes to send the communication packet identifying a flagged OM. The method 930 may include, at block 948, communicating the assembled communication packet to one or more smart nodes in the imminent path of the flagged OOI or adjacent nodes. Accordingly, the smart nodes in the path of the flagged OOI may keep the emergency service notified of the current position of the flagged OOI.
The vehicle 105 also may include various sensors that operate to gather information about the environment in which the vehicle is traveling. These sensors may include, for example: a location sensor 1060 such as a GPS device; object detection sensors such as one or more cameras 1062; a LiDAR sensor system 1064; and/or a radar and or and/or a sonar system 1066. The sensors also may include environmental sensors 1068 such as a precipitation sensor and/or ambient temperature sensor. The object detection sensors may enable the vehicle 105 to detect objects that are within a given distance or range of the vehicle 105 in any direction, while the environmental sensors collect data about environmental conditions within the vehicle's area of travel. The system architecture 1000 will also include one or more cameras 1062 for capturing images of the environment.
During operations, information is communicated from the sensors to an on-board computing device 1010. The on-board computing device 1010 analyzes the data captured by the sensors and optionally controls operations of the vehicle based on results of the analysis. For example, the on-board computing device 1010 may control braking via a brake controller 1022; direction via a steering controller 1024; speed and acceleration via a throttle controller 1026 (in a gas-powered vehicle) or a motor speed controller 1028 (such as a current level controller in an electric vehicle); a differential gear controller 1030 (in vehicles with transmissions); and/or other controllers such as an auxiliary device controller 1054. The on-board computing device 1010 may include an autonomous vehicle navigation controller 1020 configured to control the navigation of the vehicle through an intersection, as will be described in more detail in relation to
Geographic location information may be communicated from the location sensor 1060 to the on-board computing device 1010, which may then access a map of the environment that corresponds to the location information to determine known fixed features of the environment such as streets, buildings, stop signs and/or stop/go signals. Captured images from the cameras 1062 and/or object detection information captured from sensors such as a LiDAR system 1064 is communicated from those sensors) to the on-board computing device 1010. The object detection information and/or captured images may be processed by the on-board computing device 1010 to detect objects in proximity to the vehicle 105. In addition or alternatively, the vehicle 105 may transmit any of the data to a remote server system 155 (
The navigation controller 1020 may include a traffic light control engine (TLCE) 1115 interfaced with the navigation engine 1021. The traffic light control engine 1115 may be implemented using hardware, firmware, software or a combination of any of these. For instance, the traffic light control engine 1115 may be implemented as part of a microcontroller, processor, and/or GPU. The traffic light control engine 1115 may include or interface with a register and/or data store for storing data and programming instructions, which when executed, performs traffic light classification based on processed sensor information, such as computer vision system information, associated with an intersection being approached by a vehicle 105.
The traffic light control engine 1115 may include a traffic light tracker 1117 and a traffic light classifier 1119. According to various embodiments, a digital image such as a raster image may be captured by the vehicle CVS 115 (
The signal elements 135 may include circular lights and arrow lights. However, the features of each of the signal elements 135 may be any of various signal element features such as, for example, a green light, a yellow light, a red light, a circular light, a left arrow light, a right arrow light, an light having an arrow positioned in an arbitrary direction, a forward arrow light, a flashing yellow light, a flashing red light, a U-turn light, a bicycle light, an X-light, and/or any other suitable traffic signal element features. It is further noted that the traffic signal device 130 may include any suitable number of signal elements 135, having various positions on the face of the traffic signal device 130. The traffic signal elements 135 correspond to a designated light fixture configured to transmit traffic instructions to one or more drivers. The classification state of the traffic signal face includes a classification state based on one or more operational states of: a green light state; a yellow light state; a red light state; a circular light state; a left arrow light state; a right arrow light state; a forward arrow light state; a flashing yellow light state; and a flashing red light state. The traffic light classifier 1119 receives digital image data which is subsequently processed using an image processing system (IPS) such as described in
The traffic light tracker 684 may track the traffic light of an imminent intersection for a duration of time until the vehicle passes through the intersection by tracking the classification state of the traffic light. The tracked classification states of a traffic light by the vehicle in-range may change one or more times before the vehicle has successfully passed through the traffic light. The term “passed through the traffic light” includes, performing one of by the vehicle, turning right or left at a traffic light intersection and driving straight through the intersection without the need to make a turn.
The navigation controller 1020 may include an object-based control engine (OBCE) 1125 interfaced with the navigation engine 1021. The object-based control engine 1125 may be implemented using hardware, firmware, software or a combination of any of these. For instance, the object-based control engine 1125 may be implemented as part of a microcontroller, processor, and/or GPU. The object-based control engine 1125 may include or interface with a register and/or data store for storing data and programming instructions, which when executed, performs object detection based on processed sensor information, such as computer vision system information and track stationary and moving objects, vehicles and actors to and along an imminent path to be driven by the vehicle 105.
The object-based control engine 1125 may include an image signal processor 1127, OOI detector 1129, OOI classifier 1130 and an OOI tracker 1131. The object-based control engine 1125 may include an OOI motion forecaster 1135. The OOI detector 1129, OOI tracker 1131 and OOI motion forecaster 1135 operate in a similar manner as the smart node as described in relation to
The object-based control engine 1125 may include a processor, such as a GPU for performing the processing channel for each camera of the vehicle CVS and this GPU may be part of the navigation controller 1020. In other words, in some embodiments, the vehicle CVS and the navigation controller may share a processor.
The object-based control engine 1125 may be used during the operation of the vehicle 105 such that OOIs captured along a driven portion of the imminent path are extracted, identified, classified, located, and the motion of the OOI is forecasted to avoid collusion of the vehicle 105 with any of the OOIs and control the navigation of the vehicle. An OOI may be determined to be stationary or have zero motion and zero direction. For example, assume that the OOI data from the object-based control engine 1125 may be merged with APD 175 of each OOI detected by an in-range smart node 170 for operation of the navigation engine 1021 in an augmented perception mode, if available. Otherwise, the navigation engine 1021 may be operated in a non-augmented perception mode such as if augmented perception data is not available.
The navigation controller 1020 may include an APD fusion engine (APDFE) 1140 to fuse OOI data from the object-based control engine 1125 with the received APD 175 of at least one in-range smart node 170 (
The navigation controller 1020 may include a motion planning engine (MPE) 686 and a path follower engine (PFE) 1146 each of which includes machine learning algorithms for planning the motion of the vehicle based on various parameters of a to be followed path along a planned route from an origination location to a destination location of global coordinate system. The parameter may include, without limitation, motor vehicle operation laws of a jurisdiction (i.e., speed limits), objects in or surrounding a path of the vehicle, scheduled or planned route, traffic lights of intersections, and/or the like. The smart routing engine 682 may interface with the motion planning engine 686 and path follower engine 1146 so that the next motion control instructions to be executed is updated based on current traffic conditions, such as traffic congestion, or fleet control information. Motion planning control generated by the motion planning engine 686 may control acceleration, velocity, braking, and steering of the vehicle to avoid a collision at an intersection or as a vehicle travels along an imminent path.
The motion planning engine 686 and a path follower engine 1146 may also be implemented using hardware, firmware, software or a combination of any of these and interfaced with the navigation engine 1021. For instance, the motion planning engine 686 and a path follower engine 1146 may be implemented as part of a microcontroller or processor. The motion planning engine 686 may include or interface with a register and/or data store for storing data and programming instructions, which when executed, performs motion planning of the vehicle in route. The path follower engine 1146 may include or interface with a register and/or data store for storing data and programming instructions, which when executed, performs path following of a planned route to be driven by the vehicle. The imminent path may identify the planned route yet to be taken or driven from an origination location to a destination location according to a global coordinate system. The imminent path of the planned route may be updated based on, for example, traffic conditions, road closures, or emergency situations. The motion planning serves to direct the vehicles operation at every instance along an imminent path being followed. If a planned route is updated, the imminent path may be part of the planned route or the updated route.
However, the fusion engine 1140 may determine that all APD is needed for the vehicle. For example, the fusion engine 1140 may extract the location data of any OOI associated with APD 175 to determine if any APD represents an OOI which is hidden from the vehicle CVS 115 of a vehicle. In other words, receiving location coordinates of an existing OOI which is not also being tracked by the object-based control engine 1125 may cause the APD information from at least one smart node to be flagged for incorporation into the machine learning algorithms of the motion planning engine 686 to which plans ahead the motion (i.e., speed and direction) of the vehicle. In some embodiments, OOIs with matching location coordinates already being tracked by an approaching vehicle may be sorted or removed from fusion to adjust the motion of the vehicle determined in part by motion parameters under the control by the motion planning engine 686, the navigation under the control by the smart routing engine 682, or the imminent path to be travelled under the control of the path follower engine 1146.
The APD 175 may include forecasted motion information which is not available to the vehicle which may be extracted by the fusion engine 1140. In a previous example, an OOI may be a stationary vehicle which may be forecasted to imminently move or transition to a moving vehicle. Thus, the fusion engine 1140 may extract any APD 175 associated with an OOI associated which is forecasted to move or transition to a moving object or vehicle. In another example, the fusion engine 1140 may determine that the determined coordinates of an OOI at the vehicle may be different from the coordinates determined by the smart node. Accordingly, the fusion engine 1140 may update its coordinate of the OOI as determined by the vehicle with the coordinates determined by the smart node, if appropriate. Still further, the fusion engine 1140 may determine that other APD information from a node associated with an OOI is different from information derived by the vehicle's own classification. The fusion engine 1140 may fuse the APD information associated with a node with information derived by the on-board processing at the vehicle. These examples are not intended to be an exclusive list of examples of data fusion. As discussed previously, the APD information may fused from more than one smart node. As a vehicle approaches an intermediate intersection without a smart node, certain APD information of nodes in the smart node network 800 may be fused to aid the vehicle passing through the intermediate intersection.
In some embodiments, the APD information received, by the navigation engine, may be directed to and used by the motion planning engine 686, the smart routing engine 682 or the path follower engine 1146 such that offsets in location data or forecasted motion (i.e., speed and direction) may be adjusted as appropriately for use in the machines learning algorithms of the engines 686, 682 and 1146, for example.
In the various embodiments discussed in this document, the description may state that the vehicle or on-board computing device of the vehicle may implement programming instructions that cause the on-board computing device of the vehicle to make decisions and use the decisions to control operations of one or more vehicle systems. However, the embodiments are not limited to this arrangement, as in various embodiments the analysis, decision making and or operational control may be handled in full or in part by other computing devices that are in electronic communication with the vehicle's on-board computing device. Examples of such other computing devices include an electronic device (such as a smartphone) associated with a person who is riding in the vehicle, as well as a remote server system that is in electronic communication with the vehicle via a wireless communication network. The processor of any such device may perform the operations that will be discussed below.
The method 1200 may include, at block 1208, receiving at least one node network traffic information, such as from a remote server system 155 or remote processing system 650. Thus, received network traffic information, associated with nodes, may be used to control the navigation of the vehicle by updating the imminent path, for example, which effectuates changes to the path following instructions and motion planning instructions to control the navigation of the vehicle by the navigation controller. The remote server system 155 or remote processing system 650 may be configured to determine a traffic congestion condition at least one node along the imminent path based on received video stream representative of the network traffic information of the node and APD information from at least one node.
At block 1210, the method may perform object-based detection. At block 1214, the method may include classifying a TL classification state, if the image data is representative of a traffic light face. The TL classification state is sent to block 1202. The image data may include OOI and/or traffic light image data captured by the vehicle CVS 115. The OOI is meant to represent objects in the environment other than traffic lights. At any intersection, motion through the intersection may be based one at least one of the TL classification state and at least one detected OM. For example, although the vehicle may have the right of way based on the TL classification state, the vehicle may be required to yield to a pedestrian in a cross-walk, by way of non-limiting example, in the imminent path of the vehicle.
The method 1200 may include, at block 1216, receiving APD information from at least one in-range node 170 via the remote server system 155 or remote processing system 650. The method 1200 may include, at block 1218, fusing the received APD information with the OOI detected at block 1210 other than traffic light face data. The fused data from block 1218 may be used in control of the navigation of the vehicle 105.
The method 1200 may include, at block 1228, receiving traffic control information. The operations of block 1214 may use traffic control information, received at block 1228 from the gateway server 660. The traffic control information may include information associated with TL states. Block 1202 may receive traffic control information, such as TC information from the gateway server 660.
An optional display interface 1330 may permit information from the bus 1300 to be displayed on a display device 1335 in visual, graphic or alphanumeric format, such on an in-dashboard display system of the vehicle. An audio interface and audio output (such as a speaker) also may be provided. Communication with external devices may occur using various communication devices 1340 such as a wireless antenna, a radio frequency identification (RFID) tag and/or short-range or near-field communication transceiver, each of which may optionally communicatively connect with other components of the device via one or more communication system. The communication device(s) 1340 may be configured to be communicatively connected to a communications network, such as the Internet, a local area network or a cellular telephone data network.
The hardware may also include a user interface sensor 1345 that allows for receipt of data from input devices 1350 such as a keyboard or keypad, a joystick, a touchscreen, a touch pad, a remote control, a pointing device and/or microphone. Digital image frames also may be received from a camera or image capture device 1320 that can capture video and/or still images. The system also may receive data from a motion and/or position sensor 1370 such as an accelerometer, gyroscope or inertial measurement unit. The system also may receive data from a LiDAR system 1360 such as that described earlier in this document.
Web-based servers may have running thereon web-server applications stored in memory. Any servers described in this document may include multiple servers implemented in a web-based platform. In some scenarios, the servers may provide cloud based computing.
The method 1400 may include, at block 1406, classifying each OOI of the first set. The classification may be performed by the deep machine learning network, as well. The classifying may include a support vector machine (SVM) mode, or deep neural networks or other computer vision tool. For example, an OOI may be a moving vehicle, a moving object, a stationary vehicle, stationary object, moving actors and stationary actors at any instantiation along a path of the vehicle. Classifying different classes of OOIs of the first set allows various aspects of motion for each OOI to be determined, such as, location, heading, velocity and other relevant information for each classified object of interest.
The method 1400 may include, at block 1408, for each OOI of the first set, forecasting by the vehicle 105 the classified OOI's motion, the motion of an OOI may include a direction of motion. The method 1400 may include, at block 1410, querying the tracker fusion management server 655 (
The method 1400 may include, at block 1412, receiving the APD 175 of each OOI of the second set from the tracker fusion management server 655. The information associated with the APD 175 of any one OOI in the environment may include the OOI classification, the OOI forecasted motion (including direction), OOI current speed, and/or OOI location data. The APD information may include global coordinates. As previously described, depending on the intersection, the APD of the second set from the tracker fusion management server 655 may be derived from multiple smart nodes which may have adjacent or partially overlapping node VRs.
The method 1400 may include, at block 1414, fusing the APD 175 associated with each OOI of the second set from the tracker fusion management server 655 with the OOI data of the first set. The method 1400 may include, at block 1416, communicating to the gateway 660 (
The method 1400 may include, at block 1420, based on the fused OOI data of the first set and the second set and/or the traffic condition and traffic light states, controlling the motion of the vehicle along the imminent path including navigation through an intersection with a traffic signal. In other scenarios, the intersection may not include a traffic light. Hence, at least the motion planning engine of the vehicle would still use the fused OOI data of the first set from the vehicle and the OOI data of the second set from at least one smart node to control the motion of the vehicle. From block 1420, the method may return to block 1402. As the vehicle 105 travels in a direction toward the next intersection with or without a smart node, the vehicle 105 may query the tracker fusion management server 655 for the current APD associated with the next intersection or smart node.
The method 1500 may include, at block 1514, storing the external traffic control information in at least one memory location of or at least one database in memory 673 at the second server. The method 1500 may include, at block 1516, receiving a query from one or more vehicles 105 at the gateway server 660 (
The method 1500 may include, at block 1518, generating traffic control information, such as at least one of TC information and TL states. The TC information and/or TL states may be based on the query from the vehicle. The query may ask or search for current traffic control information, such as one of current TC information and/or TL states for an imminent path or a longer path which includes the imminent path. The search may generate resultant traffic control information, resultant TC information or resultant TL states. In various embodiments, the vehicle may communicate a single query for both TC information and TL states for the planned route. In various embodiments, the vehicle may communicate separate queries, such as a TL state's query and a TC query for one of the planned route or imminent path. The TC information may be stored in a database for TC information. The TL states may be stored in a database for TL states. The databases may the combined, linked or separate.
The method 1500 may include, at block 1520, communicating traffic control information, such as at least one of the TC information and TL states, to the vehicle 105 in response to the query or queries. The traffic control information may be sent via the gateway server 660 to the smart route engine 682. The TC information and/or TL states may serve to cause the planned route of a vehicle 105 to be adjusted or updated. The TL states may be used to train the TL classifier with the current TL states for any intersection with a traffic light in the imminent path. The method from block 1520 may loop back to block 1504. The traffic control information may be retrieved by the gateway server from the second server 670 and/or accessed from the traffic control infrastructure 679.
Referring now to
The OOI-APD information 1727 of the first OOI may include global coordinates 1732, OOI type 1734, OOI flag data 1736, and OOI classification data 1738, by way of non-limiting example. A type of OOI may be a vehicle, an object and actor at any instantiation of monitoring. A classified OOI may be a moving vehicle, a moving object, a moving actor, a stationary vehicle, a stationary object, and a stationary actor at any instantiation of monitoring, for example. Multiple OOIs may be further grouped together into a group type which may then be classified. The global coordinates 1732 may include information representative of a longitude coordinate 1750, a latitude coordinate 1752 and an altitude coordinate 1754. The OOI flag data 1736 may include information representation of whether a flag is set or not set. Furthermore, the situational flag (i.e., OOI flag data 1736) may include other information indicative of a type of situational flag to provide situational awareness.
The smart node record 1620 may include a plurality of current APD records from 17251 . . . 1725z where Z is a non-zero integer. Some OOIs may move out of range of the smart node 170. Accordingly, the smart node record 1620 is updated to remove OOI-APD information for any OOI that has moved out-of-range of the smart node 170. In some scenarios, at any instantiation, a smart node 170 may have no OOIs in their range. Accordingly, the current APD record may include no OOI-APD entries.
The smart node record 1620 may include information associated with the smart node's global coordinates 1760. The global coordinates in some instances may include the coordinates in the vision field of the node VR. The smart node record 1620 may include information associated with the smart node's identification 1765. The smart node identification 1765 may include at least one of an internet protocol (IP) address information, and a media access control (MAC) address information. The smart node record 1620 may include security information. The smart node record 1620 may include information including global coordinates of the vision field associated with adjacent/overlapping intersections with smart nodes 1772 and/or without smart nodes 1774.
Returning again to
The tracker fusion management server 655 may include an imminent smart node search engine 1640 to develop a search request for the APD information stored in memory 656 from one or more proximate smart nodes, for example, associated with the extracted vehicle path information. There term “proximate” may be a function of a comparison of the smart node coordinates 1760 and the vehicle's current coordinates and/or coordinates along the imminent path. In some scenarios, the vehicle may have a list of smart nodes and the related coordinates. Accordingly, in some scenarios, the vehicle may perform on-board processing to determine that one or more smart nodes are imminent along the imminent path and provide information. The query sent to the tracker fusion management server 655 from the vehicle may include information in the query associated with a smart node as determined by the vehicle. For example, assume that the L is equal to 100. Then, the query may request information for the next 100 meters of the planned route. The “next 100 meters” corresponds to the current imminent path. The query may indicate an intermediate point of origination and an intermediate point of destination. The distance between intermediate reference points, denoted as the intermediate point of origination and the intermediate point of destination, correspond to the distance of the imminent path. The vehicle may receive available APD information within that imminent path associated with the intermediate reference points. The vehicle may receive the APD information of the imminent path while at a location preceding the imminent path, partly in the imminent path from the perspective of the intermediate point of origination and any location along the path until the vehicle has cleared the intermediate point of destination.
The tracker fusion management server 655 may include a fusion APD module 1645 to fuse together APD information from at least one smart node identified in the search results. The fusion APD module may communicate, via communication device 1650, a portion of the APD information from any one smart node record, in some scenarios. For example, some stationary objects identified at an intersection on a path leading to the intersection that will not be traveled by the vehicle. Such APD information may be omitted from the APD information sent to the vehicle from tracker fusion management server 655. In other examples, moving objects or vehicles already heading away from the intersection along a path not to be traveled by the vehicle may be omitted from the APD information sent to the vehicle from the tracker fusion management server 655. On the other hand, moving objects or vehicles heading in the direction of an imminent intersection (relative to the vehicle's position) without a smart node, may be sent to the vehicle from the tracker fusion management server 655. By way of non-limiting example, APD information may be determined based on the vehicle path information extracted from the query by the tracker fusion management server 655. The tracker fusion management server 655 may use global coordinates of an imminent path, current global coordinates of the vehicle to extract APD information of OOIs to be encountered in the planned route of the vehicle. In the scenarios where the vehicle includes a list of smart nodes, the query processing engine 1625 may include an optional smart node extraction module 1630, denoted in dashed lines, to extract from the query information associated with those smart nodes along the imminent path for which the query was generated. The smart nodes may be associated with identifiers and coordinates.
The master smart node record 1820 may include current TL states 1926 received from the traffic control infrastructure 679 via the gateway 660. The TL states 1926 may be used to predict whether there is or will be traffic congestion at a particular smart node. The master smart node record 1820 may include intersection speed limits 19301 . . . 1930T where T is a non-zero integer. Each intersection speed limit 19301 . . . 1930T may differ per road merging into or exiting from an intersection. In some scenarios, an intersection may include cross roads leading into and/or away from the intersection. The master node record 1820 may also be periodically sent to or queried by the vehicle such as for initial planning of the planned route. For example, a query for information in L meters may include different lengths for different routes between reference points such as a point of origination and a point of destination. The updated master smart node records may be helpful in planning an end to end route. However, the “L” may be smaller after a planned route is selected. The “L” in meters, for example, may be the distance for a next imminent path in the planned route. Information of the master smart node record 1820 may be stored by and determined by the vehicle 105.
Returning again to
The traffic condition prediction module 1840 may include a speed extractor 1844 to determine a speed of each moving vehicle OOI passing in any direction of the intersection. The traffic condition prediction module 1840 may include a speed limit extractor 1846 to obtain the current speed limit for a particular prediction instantiation. The traffic condition prediction module 1840 may include an adjacent intersection congestion estimating engine 1848.
The route control module 1830 may use TL states to synchronize the received APD information to at least one TL state cycle. A TL state cycle may include an ordered arrangement of red, green, and yellow light phases, by way of non-limiting example. Each phase may have a particular duration. In some scenarios, the traffic condition prediction module 1840 may extract the APD information captured during a green phase of at least one TL state cycle to determine a level of traffic congestion at a node. During the green phase of a TL state cycle, in general, vehicles are expected to be traveling through an intersection at speeds generally close to the road's speed limit if there is little to no traffic congestions at the intersection.
The traffic condition prediction module 1840 may use the intersection speed limits 19301 . . . 1930T to predict or determine whether there is or will be traffic congestion at a particular smart node. In some scenarios, the vehicle may have speed limits of intersections, roads and other highways already stored for use by the vehicle navigation controller 1020. For example, a traffic congestion condition may be determined by the traffic condition prediction module 1840 based on speeds of moving vehicle OOIs being below an intersection speed limit in a particular direction such as during a green phase of the TL states 1926 of the intersection. In other embodiments, the traffic condition prediction module 1840 may track the moving vehicle OOIs to determine whether the tracked vehicle OOIs remain nearly stationary in the path for an amount of time. The amount of time may be based on whether two or more TL state cycles have been completed with limited advancement in the predicted direction of motion of the same vehicle OOIs along the path. A moving vehicle OOI is tracked based on the OOI-APD information in the smart node record.
The master smart node 1820 data may be received by the vehicle 105 from the remote server system 155. The data of the master smart node 1820 may represent network traffic information representative of a traffic condition detected within the vision range of the node and within vision ranges of a plurality of additional nodes. In this context, “network traffic information” is associated with traffic information derived from the network 800 of nodes. The “network traffic information” is different from the “traffic control information” from the traffic control infrastructure 679 associated with a traffic control system.
In other scenarios, the speed extractor 1844 of the traffic condition prediction module 1840 may identify parameters such as an amount of speed and/or speed thresholds below the intersection speed limit to detect a traffic congestion condition. Since the speeds may be changed based on a time of day, the speed extractor 1844 may track and/or update the speed thresholds for any prediction instantiation before making comparisons to determine if a vehicle OOI is moving very slow or forced to be nearly stationary, for example. In another example, in school zones, the speed limit is lowered during certain times of day. In certain highways, the speed limits may be changed based on construction or for other reasons. The term “nearly stationary” may include speeds of 5 miles per hour (MPH) or less. The term “nearly stationary” may include speeds of 10 MPH or less. Depending on the intersection, the term “nearly stationary” may include speeds of 20 MPH or less, for example.
The adjacent intersection congestion estimating engine 1848 may estimate an imminent traffic congestion condition based on traffic congestion conditions of adjacent intersection or adjacent intersections with smart nodes in the network 800. The adjacent intersection congestion estimating engine 1848 may estimate imminent traffic congestion at one or more adjacent intersections that are adjacent to a respective one smart node. For example, traffic condition prediction module 1840 may predict traffic congestion leading away from a smart node based on the time (i.e., two or more TL state cycles) a moving vehicle OOI stays in a turning lane. If a vehicle is predicted to turn left or right but remains stationary or moves at a very slow speed in a turning lane for several TL state cycles, such indication may be representative of a traffic congestion condition at an adjacent intersection or adjacent smart node in the direction of the turn.
The route control module 1830 may include a node traffic delay time calculator 1850. The node traffic delay time calculator 1850 may estimate an amount of delay of vehicle may experience traveling through a node at a particular instantiation. A node traffic delay time may be determined relative to the stored speed limit for each direction to and from the node and the time for a moving vehicle to travel to, through and from the intersection of the node. Each direction of travel to and from the intersection with a smart node may be have its own delay time metric. The smart node delay time metric 19401 . . . 1940T for each direction may be stored in the master smart node record 1820. As the value of the node traffic delay time increases, such increase may be a metric to predict an imminent traffic congestion levels.
The route control module 1830 may include a route generator engine 1855. In the event of an accident, for example, traffic congestion may develop at a location of the accident. The traffic congestion levels may be determined by the traffic condition prediction module 1840 at a particular location of a smart node. The route generator engine 1855 may include a shortest travel-time path finder 1860 to find one or more paths through the network of smart nodes and/or intersections without a smart node based on the calculated smart node delay time metrics 19401 . . . 1940T stored in the master smart node record 1820. The shortest travel-time path may estimate the travel-time based on current traffic congestion conditions through the nodes in combination with a length of the path, speed limit and delay times. The route generator engine 1855 may generate a new route of global coordinates for the remaining planned route based on a selected shortest travel-time path. The new route may be determined by the smart route engine 682 of the vehicle to update the planned route.
The route control module 1830 may find a new route for other reasons then traffic congestion. The route control module 1830 may need to change a destination location which changes the planned route. However, when scheduling a new imminent path, for example, the route control module 1830 may use a shortest travel-time path finder 1860 to find a new route using the real-time status of the traffic congestion or imminent traffic congestion. The shortest travel-time path finder 1860 may use machine algorithms that also consider at least one of tolls, highways, state laws, estimated time of arrival, traffic congestion trends, etc.
The above-disclosed features and functions, as well as alternatives, may be combined into many other different systems or applications. Various components may be implemented in hardware or software or embedded software. Various presently unforeseen or unanticipated alternatives, modifications, variations or improvements may be made by those skilled in the art, each of which is also intended to be encompassed by the disclosed embodiments.
Terminology that is relevant to the disclosure provided above includes:
An “automated device” or “robotic device” refers to an electronic device that includes a processor, programming instructions, and one or more components that based on commands from the processor can perform at least some operations or tasks with minimal or no human intervention. For example, an automated device may perform one or more automatic functions or function sets. Examples of such operations, functions or tasks may include without, limitation, navigation, transportation, driving, delivering, loading, unloading, medical-related processes, construction-related processes, and/or the like. Example automated devices may include, without limitation, autonomous vehicles, drones and other autonomous robotic devices.
The term “vehicle” refers to any moving form of conveyance that is capable of carrying either one or more human occupants and/or cargo and is powered by any form of energy. The term “vehicle” includes, but is not limited to, cars, trucks, vans, trains, autonomous vehicles, aircraft, aerial drones and the like. An “autonomous vehicle” is a vehicle having a processor, programming instructions and drivetrain components that are controllable by the processor without requiring a human operator. An autonomous vehicle may be fully autonomous in that it does not require a human operator for most or all driving conditions and functions, or it may be semi-autonomous in that a human operator may be required in certain conditions or for certain operations, or that a human operator may override the vehicle's autonomous system and may take control of the vehicle. Autonomous vehicles also include vehicles in which autonomous systems augment human operation of the vehicle, such as vehicles with driver-assisted steering, speed control, braking, parking and other systems.
In this document, the terms “street,” “lane” and “intersection” are illustrated by way of example with vehicles traveling on one or more roads. However, the embodiments are intended to include lanes and intersections in other locations, such as parking areas. In addition, for autonomous vehicles that are designed to be used indoors (such as automated picking devices in warehouses), a street may be a corridor of the warehouse and a lane may be a portion of the corridor. If the autonomous vehicle is a drone or other aircraft, the term “street” may represent an airway and a lane may be a portion of the airway. If the autonomous vehicle is a watercraft, then the term “street” may represent a waterway and a lane may be a portion of the waterway.
As used in this document, the term “light” means electromagnetic radiation associated with optical frequencies, e.g., ultraviolet, visible, infrared and terahertz radiation. Example emitters of light include laser emitters and other emitters that emit converged light. In this document, the term “emitter” will be used to refer to an emitter of light, such as a laser emitter that emits infrared light.
An “electronic device” or a “computing device” refers to a device that includes a processor and memory. Each device may have its own processor and/or memory, or the processor and/or memory may be shared with other devices as in a virtual machine or container arrangement. The memory will contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations according to the programming instructions.
The terms “memory,” “memory device,” “data store,” “data storage facility” and the like each refer to a non-transitory device on which computer-readable data, programming instructions or both are stored. Except where specifically stated otherwise, the terms “memory,” “memory device,” “data store,” “data storage facility” and the like are intended to include single device embodiments, embodiments in which multiple memory devices together or collectively store a set of data or instructions, as well as individual sectors within such devices.
The terms “processor” and “processing device” refer to a hardware component of an electronic device that is configured to execute programming instructions. Except where specifically stated otherwise, the singular term “processor” or “processing device” is intended to include both single-processing device embodiments and embodiments in which multiple processing devices together or collectively perform a process.
The term “execution flow” refers to a sequence of functions that are to be performed in a particular order. A function refers to one or more operational instructions that cause a system to perform one or more actions. In various embodiments, an execution flow may pertain to the operation of an automated device. For example, with respect to an autonomous vehicle, a particular execution flow may be executed by the vehicle in a certain situation such as, for example, when the vehicle is stopped at a red stop light that has just turned green. For instance, this execution flow may include the functions of determining that the light is green, determining whether there are any obstacles in front of or in proximity to the vehicle and, only if the light is green and no obstacles exist, accelerating. When a subsystem of an automated device fails to perform a function in an execution flow, or when it performs a function out of order in sequence, the error may indicate that a fault has occurred or that another issue exists with respect to the execution flow.
In this document, the terms “communication link” and “communication path” mean a wired or wireless path via which a first device sends communication signals to and/or receives communication signals from one or more other devices. Devices are “communicatively connected” if the devices are able to send and/or receive data via a communication link. “Electronic communication” refers to the transmission of data via one or more signals between two or more electronic devices, whether through a wired or wireless network, and whether directly or indirectly via one or more intermediary devices.
An “automated device monitoring system” is a set of hardware that is communicatively and/or electrically connected to various components (such as sensors) of an automated device to collect status or operational parameter values from those components. An automated device monitoring system may include or be connected to a data logging device that includes a data input (such as a wireless receiver) that is configured to receive device operation data directly or indirectly from the device's components. The monitoring system also may include a processor, a transmitter and a memory with programming instructions. A monitoring system may include a transmitter for transmitting commands and/or data to external electronic devices and/or remote servers. In various embodiments, a monitoring system may be embedded or integral with the automated device's other computing system components, or it may be a separate device that is in communication with one or more other local systems, such as, for example in the context of an autonomous vehicle, an on-board diagnostics system.
In this document, when relative terms of order such as “first” and “second” are used to modify a noun, such use is simply intended to distinguish one item from another, and is not intended to require a sequential order unless specifically stated.
In addition, terms of relative position such as “vertical” and “horizontal”, or “front” and “rear”, when used, are intended to be relative to each other and need not be absolute, and only refer to one possible position of the device associated with those terms depending on the device's orientation. When this document uses the terms “front,” “rear,” and “sides” to refer to an area of a vehicle, they refer to areas of vehicle with respect to the vehicle's default area of travel. For example, a “front” of an automobile is an area that is closer to the vehicle's headlamps than it is to the vehicle's tail lights, while the “rear” of an automobile is an area that is closer to the vehicle's tail lights than it is to the vehicle's headlamps. In addition, the terms “front” and “rear” are not necessarily limited to forward-facing or rear-facing areas but also include side areas that are closer to the front than the rear, or vice versa, respectively. “Sides” of a vehicle are intended to refer to side-facing sections that are between the foremost and rearmost portions of the vehicle.
