Patent Application
20170344855
The presently disclosed subject matter relates to techniques of predicting traffic collisions and, more particularly, to methods and systems enabling prediction of traffic collisions between vehicles at an intersection.
A large number of traffic accidents between different vehicles occur at intersections. Some of these accidents can be avoided if a collision between two vehicles at a certain intersection can be predicted in advance, in which case a warning can be issued to one or both drivers, who can then change course.
Various different techniques exist for predicting traffic collisions between two vehicles. One particular sub-class of techniques involves utilizing motion pattern analysis to estimate the future positions of two vehicles at a certain intersection, based on observed motion patterns of other vehicles at the same intersection.
Problems of predicting traffic collisions using motion pattern analysis are known in the art, and various techniques have been developed to provide solutions. For example,
St-Aubin, P., L. F. Miranda-Moreno, and N. Saunier, “Road User Collision Prediction Using Motion Patterns Applied to Surrogate Safety Analysis”, Transportation Research Board 93rd Annual Meeting, Washington, D.C. (2014), discloses using discretized motion pattern maps for predicting potential collisions between road users by extracting road user trajectory data from video data of a traffic scene.
M. Muffert, D. Pfeiffer and U. Franke, “A stereo-vision based object tracking approach at roundabouts”, IEEE Intell, Transp, Syst, Mag., vol. 5, no. 2, pp. 22-32 (2013), discloses a stereo-vision based system for the recognition of dangerous situations at roundabouts.
K. Minoura and T. Watanabe, “Driving support by estimating vehicle behavior”, 21st International Conference on Pattern Recognition (ICPR), pp. 1144-1147 (2012), discloses predicting vehicle behaviors by using a surveillance camera and an on-board camera and applying a Hidden Markov Model (HMM) to predict straight, right turn, left turn, change to right lane, and change to left lane.
S. Atev, O. Masoud, R. Janardan, and N. Papanikolopoulos, “A collision prediction system for traffic intersections”, Proc. IEEE/RSJ Conf. Intelligent Robots and Systems (IROS) (2005), discloses monitoring traffic intersections in real-time and predicting possible collisions using three-dimensional vehicle size estimation, target localization, false-positive reduction and a collision prediction algorithm using the time-as-axis paradigm.
N. Saunier, T. Sayed and C Lim, “Probabilistic collision prediction for vision-based automated road safety analysis”, Proc. IEEE Int. Conf. Intell. Transp. Syst., pp. 872-878 (2007), discloses computing the collision probability for any two road users in an interaction by processing traffic video data, detecting and tracking the road users, and analyzing their interactions by using motion patterns to predict the road users' movements and determine their probability of being involved in a collision.
N. Saunier and T. Sayed, “Clustering vehicle trajectories with hidden Markov models application to automated traffic safety analysis”, Proc. IJCNN, pp. 4132-4138 (2006), discloses extracting traffic conflicts from video sensor data by clustering vehicle trajectories using a k-means approach with hidden Markov models, and detecting traffic conflicts by identifying and adapting pairs of models of conflicting trajectories.
B. Morris and M. Trivedi, “A survey of vision-based trajectory learning and analysis for surveillance”, IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 8, pp. 1114-1127 (2008), presents a survey of trajectory-based activity analysis for visual surveillance.
The references cited above teach background information that may be applicable to the presently disclosed subject matter. Therefore the full contents of these publications are incorporated by reference herein where appropriate for appropriate teachings of additional or alternative details, features and/or technical background.
The choices of which path to take at a given intersection can sometimes be different for different classes of vehicles. For example, a car approaching a certain intersection may be allowed to turn right, go straight or turn left, whereas a bicycle approaching the intersection from the same direction (and perhaps even the same lane) as the car may only be allowed to turn right or go straight. Given that the paths available to one class might be different than the paths available to another class, better predictions can be made about any particular vehicle's future positions if the class of vehicle and the paths available to the class are known in advance.
However, the collision prediction systems of the prior art typically do not differentiate between different classes of vehicles, such as cars and trucks, or cars and motorcycles, trucks and motorcycles, cars and bicycles, etc. Furthermore, the collision prediction systems of the prior art typically track vehicles using a camera mounted high above the intersection, e.g. to prevent problems of occlusion. From such a high vantage point, these cameras are for the most part unsuitable for detecting and tracking small, narrow objects like bicycles. Yet, accidents between cars and bicycles are some of the most common and serious accidents, as it is relatively easy for a driver of a car to miss noticing a bicycle on the road, and cyclists often sustain very serious injuries after having been hit by a car.
In accordance with certain aspects of the presently disclosed subject matter, there is provided a method of generating an intersection model useable for predicting traffic collisions at a given intersection between vehicles of different classes, the method implemented by a processing unit and comprising, by the processing unit: (a) classifying a vehicle appearing in image data as belonging to a given class out of a predefined set of classes, wherein the image data is informative of a plurality of successive images of vehicles at the intersection; (b) tracking the vehicle to extract the vehicle's trajectory and associating the extracted trajectory with the class of the vehicle; (c) repeating operations (a)-(b) in respect of a plurality of vehicles belonging to different classes to obtain a plurality of extracted trajectories, each associated with a given class out of the predefined set of classes, until a completion criterion is satisfied; (d) for each given class, generating a plurality of reference trajectories associated with the given class and with the given intersection using at least part of the plurality of extracted trajectories associated with the given class, and (e) for each given class, clustering the reference trajectories into one or more clusters, each cluster informative of a path available to be taken by vehicles of the given class at the intersection.
In accordance with certain other aspects of the presently disclosed subject matter, there is provided a system for generating an intersection model useable for predicting traffic collisions at a given intersection between vehicles of different classes, the system comprising a processing unit including at least a processor operatively coupled to a memory, the processing unit configured to: (a) classify a vehicle appearing in image data as belonging to a given class out of a predefined set of classes, wherein the image data is informative of a plurality of successive images of vehicles at the intersection; (b) track the vehicle to extract the vehicle's trajectory and associating the extracted trajectory with the class of the vehicle; (c) repeat operations (a)-(b) in respect of a plurality of vehicles belonging to different classes to obtain a plurality of extracted trajectories, each associated with a given class out of the predefined set of classes, until a completion criterion is satisfied; (d) for each given class, generate a plurality of reference trajectories associated with the given class and with the given intersection using at least part of the plurality of extracted trajectories associated with the given class, and (e) for each given class, cluster the reference trajectories into one or more clusters, each cluster informative of a path available to be taken by vehicles of the given class at the intersection.
In accordance with further aspects of the presently disclosed subject matter, and optionally in combination with other aspects, a data structure can be generated comprising data indicative of all conflicting pairs of reference trajectories, wherein the reference trajectories in a conflicting pair are associated with different classes and wherein a given pair of reference trajectories conflict when the minimal distance between the reference trajectories in the pair is less than a predefined threshold.
In accordance with further aspects of the presently disclosed subject matter, and optionally in combination with other aspects, there can be stored, in the data structure, data informative of one or more pairs of conflicting data points for each pair of conflicting reference trajectories, each one or more pairs of conflicting data points constituted by a first conflicting data point in the first reference trajectory and a second conflicting data point in the second reference trajectory.
In accordance with certain other aspects of the presently disclosed subject matter, there is provided a method of determining the likelihood of a traffic collision between vehicles of different vehicle classes at an intersection, the method implemented by a processing unit and comprising: obtaining data informative of an intersection model associated with the intersection, the intersection model comprising a plurality of reference trajectories associated with each of a plurality of predefined vehicle classes, the intersection model further comprising conflict data informative of conflicting reference trajectories associated with different classes; classifying a first vehicle appearing in image data as belonging to a first class out of the plurality of predefined vehicle classes, and classifying a second vehicle appearing in the image data as belonging to a second class out of the plurality of predefined vehicle classes different from the first class, wherein the image data is informative of a plurality of successive images of vehicles at the intersection; tracking the first and second vehicle using the image data to extract a first trajectory associated with the first vehicle and a second trajectory associated with the second vehicle; selecting, from the plurality of reference trajectories comprised in the intersection model and associated with the first class, a first set of reference trajectories best matching the first trajectory, and selecting, from the plurality of reference trajectories comprised in the intersection model and associated with the second class, a second set of reference trajectories best matching the second trajectory; identifying, using the data indicative of the intersection model, one or more pairs of conflicting reference trajectories, each pair constituted by a first reference trajectory from the first set and a conflicting second reference trajectory from the second set; for each pair of conflicting reference trajectories, mapping the first vehicle to a data point on the first reference trajectory in accordance with the first trajectory, and mapping the second vehicle to a data point on the second reference trajectory in accordance with the second trajectory; generating data indicative of a likelihood of a collision between the first and second vehicle in accordance, at least, with the data points to which the vehicles have been, respectively, mapped; and generating a warning when the generated data satisfy a predetermined criterion.
In accordance with certain other aspects of the presently disclosed subject matter, there is provided a system for determining the likelihood of a traffic collision between vehicles of different vehicle classes at an intersection, the system comprising a processing unit including at least a processor operatively coupled to a memory, the processing unit configured to: obtain from the memory data informative of an intersection model associated with the intersection, the intersection model comprising a plurality of reference trajectories associated with each of a plurality of predefined vehicle classes, the intersection model further comprising conflict data informative of conflicting reference trajectories associated with different classes; classify a first vehicle appearing in image data as belonging to a first class out of the plurality of predefined vehicle classes, and classifying a second vehicle appearing in the image data as belonging to a second class out of the plurality of predefined vehicle classes different from the first class, wherein the image data is informative of a plurality of successive images of vehicles at the intersection; track the first and second vehicle using the image data to extract a first trajectory associated with the first vehicle and a second trajectory associated with the second vehicle; select, from the plurality of reference trajectories comprised in the intersection model and associated with the first class, a first set of reference trajectories best matching the first trajectory, and select, from the plurality of reference trajectories comprised in the intersection model and associated with the second class, a second set of reference trajectories best matching the second trajectory; identify, using the data indicative of the intersection model, one or more pairs of conflicting reference trajectories, each pair constituted by a first reference trajectory from the first set and a conflicting second reference trajectory from the second set; for each pair of conflicting reference trajectories, map the first vehicle to a data point on the first reference trajectory in accordance with the first trajectory, and map the second vehicle to a data point on the second reference trajectory in accordance with the second trajectory; generate data indicative of a likelihood of a collision between the first and second vehicle in accordance, at least, with the data points to which the vehicles have been, respectively, mapped; and generate a warning when the generated data satisfy a predetermined criterion.
In accordance with certain other aspects of the presently disclosed subject matter, there is provided a non-transitory storage medium comprising instructions that when executed by a processing unit comprising at least a processor operatively coupled to a memory, cause the processing unit to: obtain data informative of an intersection model associated with the intersection, the intersection model comprising a plurality of reference trajectories associated with each of a plurality of predefined vehicle classes, the intersection model further comprising conflict data informative of conflicting reference trajectories associated with different classes; classify a first vehicle appearing in image data as belonging to a first class out of the plurality of predefined vehicle classes, and classifying a second vehicle appearing in the image data as belonging to a second class out of the plurality of predefined vehicle classes different from the first class, wherein the image data is informative of a plurality of successive images of vehicles at the intersection; track the first and second vehicle using the image data to extract a first trajectory associated with the first vehicle and a second trajectory associated with the second vehicle; select, from the plurality of reference trajectories comprised in the intersection model and associated with the first class, a first set of reference trajectories best matching the first trajectory, and select, from the plurality of reference trajectories comprised in the intersection model and associated with the second class, a second set of reference trajectories best matching the second trajectory; identify, using the data indicative of the intersection model, one or more pairs of conflicting reference trajectories, each pair constituted by a first reference trajectory from the first set and a conflicting second reference trajectory from the second set; for each pair of conflicting reference trajectories, map the first vehicle to a data point on the first reference trajectory in accordance with the first trajectory, and map the second vehicle to a data point on the second reference trajectory in accordance with the second trajectory; generate data indicative of a likelihood of a collision between the first and second vehicle in accordance, at least, with the data points to which the vehicles have been, respectively, mapped; and generate a warning when the generated data satisfy a predetermined criterion.
In accordance with further aspects of the presently disclosed subject matter, and optionally in combination with other aspects, generating data indicative of a likelihood of a collision between the first and second vehicle in accordance with the data points to which the vehicles have been, respectively mapped can comprise determining, using the intersection model, a first conflicting data point on the first reference trajectory and a second conflicting data point on the second reference trajectory; determining, in accordance with an index of the data point on the first conflicting reference trajectory to which the first vehicle has been mapped and an index number of the first conflicting data point and a frame rate of the image data, a time to arrival of the first vehicle to the first conflicting data point; and determining, in accordance with an index of the data point on the second conflicting reference trajectory to which the second vehicle has been mapped and an index number of the second conflicting data point and a frame rate of the image data, a time to arrival of the second vehicle to the second conflicting data point.
In accordance with further aspects of the presently disclosed subject matter, and optionally in combination with other aspects, each reference trajectory comprised in the intersection model and associated with a given class can further be associated with a given path out of one or more available paths available to be taken by vehicles of the given class through the intersection; and selecting a set of reference trajectories best matching a given vehicle's associated trajectory can comprise determining one or more predicted paths out of the one or more available paths in accordance with the given vehicle's associated trajectory; assigning a matching cost to each reference trajectory associated with each predicted path in accordance with the given vehicle's trajectory; and for each one or more predicted paths, selecting the reference trajectory associated with the predicted path having the lowest matching cost as between all other reference trajectories also associated with the predicted path.
In accordance with further aspects of the presently disclosed subject matter, and optionally in combination with other aspects, the probability can be determined of, for at least one identified pair of reference trajectories, the first vehicle taking the path p associated with the first trajectory and the second vehicle taking the path q associated with the second reference trajectory, and the predetermined criterion can be at least partially met when the probability of the first and second vehicle taking the pair of paths p,q, respectively, is greater than the probability of the first and second vehicles taking a different pair of paths.
In accordance with further aspects of the presently disclosed subject matter, and optionally in combination with other aspects, one of the first and second vehicle can be a car and the other of the first and second vehicle can be a bicycle.
Among advantages of certain embodiments of the presently disclosed subject matter is the ability to separately track different classes of vehicles, and generate warnings when there is a likelihood of collision between vehicles of the different classes. The invention is especially suitable for predicting collisions between cars and bicycles.
In order to understand the invention and to see how it can be carried out in practice, embodiments will be described, by way of non-limiting examples, with reference to the accompanying drawings, in which:
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the presently disclosed subject matter may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the presently disclosed subject matter.
Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “processing”, “classifying”, “associating”, “comparing”, “generating”, “mapping”, “tracking”, “obtaining”, “extracting”, “determining” “selecting” or the like, refer to the action(s) and/or process(es) of a computer that manipulate and/or transform data into other data, said data represented as physical, such as electronic, quantities and/or said data representing the physical objects. The term “computer” should be expansively construed to cover any kind of hardware-based electronic device with data processing capabilities including, by way of non-limiting example, the processing unit disclosed in the present application.
The terms “non-transitory memory” and “non-transitory storage medium” as used herein should be expansively construed to cover any volatile or non-volatile computer memory suitable to the presently disclosed subject matter.
The operations in accordance with the teachings herein may be performed by a computer specially constructed for the desired purposes or by a general-purpose computer specially configured for the desired purpose by a computer program stored in a non-transitory computer-readable storage medium.
The term “criterion” used in this patent specification should be expansively construed to include any compound criterion, including, for example, several criteria and/or their logical combinations.
Embodiments of the presently disclosed subject matter are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the presently disclosed subject matter as described herein.
Bearing this in mind, attention is drawn to
In certain embodiments, camera (12) is mounted between 1-5 meters above the ground in close proximity to a given intersection and aimed at the intersection in such a manner so as to generate image data informative of vehicles approaching the intersection (e.g. at least 25 meters before the intersection, or between 30-50 meters before the intersection) and at the intersection. As used herein, the term “intersection” should be expansively construed to cover a section of roadway in which a vehicle can take different routes. Examples of different routes include turning right, turning left, continuing straight, etc. It should be appreciated that a single route can encompass one or several lanes.
In certain embodiments, processing unit (14) includes a memory (16) and a processor (20) operatively coupled e.g. via a communication bus (22). Optionally, processing unit (14) can further include an input/output (I/O) interface (18) and/or a communication interface (24) operatively coupled to the processor and memory, e.g. via communication bus (22). Each interface can comprise (individually or shared with other interfaces) network interface (e.g. Ethernet card), communication port, etc.
Memory (16) can be, e.g., non-volatile computer readable memory, and can be configured to store, inter alia, image data generated by camera (12), data generated by processor (20), and/or program instructions for performing functions related to predicting traffic collision.
Processor (20) is configured to provide processing necessary for collision prediction analysis as further detailed below in the flowcharts in
Processor (20) can be configured to execute several functional modules in accordance with computer-readable instructions stored on a non-transitory computer-readable storage medium. Such functional modules are referred to hereinafter as comprised (or included) in the processor. In certain embodiments, the computer-readable instructions can be stored in memory (16). Processor (20) can include, in certain embodiments, such functional modules as a classifier (26) to detect and classify a vehicle in image data, tracking module (28) to track a vehicle in image data and extract its trajectory, model generator (29) to process extracted trajectories and generate an intersection model, prediction engine (31) to calculate a likelihood of a collision between two tracked vehicles using an intersection model and to generate data indicative of same, and a warning module (32) to generate a collision warning, as will be further detailed with reference to the flowcharts in
In certain embodiments, I/O interface (18) can be configured to perform input/output operations related to predicting traffic collisions (including, e.g. receiving user-provided configuration data, and outputting test data for user verification). I/O interface (18) can be connected to at least one input device such as a keyboard (not shown) and/or at least one output device such as a display (not shown).
In certain embodiments, communication interface (24) can be configured to perform send and receive operations related to predicting traffic collisions. For example, using communication interface (24), processing unit (14) can send and receive data to/from other components of CPS which may be physically located external to processing unit (14). For example, camera (12) can be physically located at a certain intersection and configured to send image data to processing unit (14) physically located away from the intersection. Processing unit (14) can perform a collision prediction analysis using processor (20) and send the results of such analysis to a display board located at the intersection, to a computer controlling the intersection (e.g. an intelligent intersection controller in the case that the intersection is an “intelligent” intersection, as that term is used in the art), or to a vehicle approaching the intersection (e.g. in the case that the vehicle is capable of communicating with external computer systems), as will be detailed below with reference to the flowcharts in
The operation of CPS (10) will now be detailed, in accordance with certain embodiments, with reference to the flowcharts in
Processing unit (14), e.g. classifier (26), processes the image data to detect and classify (202) a vehicle appearing in the image data according to the vehicle's class based on a set of predefined classes, e.g. car, truck, bus, motorcycle, etc. In certain embodiments, the set of predefined classes includes car and bicycle. Methods of detecting and classifying objects appearing in images are known in the art. For example, a Support Vector Machine (SVM) can be used to detect and classify the objects after having been trained using Histogram of Oriented Gradients (HOG) feature descriptors extracted from labelled training images. This trained SVM model can be used to classify image regions described using HOG image descriptors. An image “descriptor” refers to a vector of values (e.g. pixel values, gradients, etc.) which describes an image patch. Image “features” refer to a group of one or more pixels in an image which are distinguishable (i.e. are visually distinct in some predetermined respect) from neighboring pixels. Harris corners are an example of image features.
Processing unit (14), e.g. tracking module (28), tracks (204) the classified vehicle through successive frames of the video and extracts data informative of the vehicle's trajectory in the image domain (hereinafter referred to as the vehicle's “trajectory”). The extracted trajectory is associated (206) with the class of the vehicle as previously classified, and stored in memory (16) in association with the class. The process of classifying (202), tracking (204) and associating (206) and storing is repeated in respect of a plurality of different vehicles of different classes until a completion criterion is satisfied (208). The completion criterion can be, e.g. the number of stored trajectories in each of a set of classes meets a predetermined threshold, or the number of tracking hours meets a predetermined threshold, or other criteria. As used herein, a completion “criterion” can be one criterion or several criteria. The completion criterion used should guarantee that upon being met, a sufficient number of stored trajectories (e.g. n≧5) for each class, in the predefined set of classes, exist for each possible route available to vehicles of the class at the intersection. Prior knowledge about the different routes available at the intersection can be used to define the completion criterion however prior knowledge is not necessary. It should be noted that in general, the more stored trajectories that are available to generate the intersection model, the more accurate the prediction is likely to be.
In certain embodiments, upon the completion criterion (208) being satisfied, processing unit (14), e.g. using model generator (29), then processes (210) the trajectories in each class to generate a set of reference trajectories in respect of the class, as will be more fully detailed below. It should be noted that, in certain embodiments, the trajectory processing (210) can, in the alternative, be performed on each trajectory as it is extracted from the image data and prior to all the trajectories having been extracted. In that case, the completion criterion is satisfied, e.g. when a sufficient number of trajectories have been extracted and processed. As a result of processing (210), the intersection is associated with a plurality of vehicle classes, each class associated with a plurality of reference trajectories r. This relationship is illustrated in
Processing unit (14), e.g. model generator (29), then clusters (212) each class's set of reference trajectories into one or more clusters, each cluster informative of a given path available to vehicles of the class at the intersection thereby learning the available paths, as will be more fully detailed below. As used herein, the term “path” is interchangeably used to refer to a path in its traditional sense (i.e. a route), and also to a cluster of reference trajectories indicative of a route available to a vehicle. As a result of clustering (212), each class's paths are learned, each path having one or more associated reference trajectories r. This relationship is illustrated in
Finally, processing unit (14), e.g. model generator (29), identifies (214) conflicting pairs of paths p, q at the given intersection, where p and q are each associated with different classes (e.g. a car and a bicycle). In certain embodiments, p and q can also be associated with the same class (e.g. two cars). As used herein, a pair of paths p, q conflict when at least one reference trajectory in p conflicts with at least one reference trajectory in q. As will be further detailed below with reference to
Tracking (204) a vehicle will now be more fully detailed in accordance with certain embodiments. Methods are known in the art for tracking objects in successive frames of image data and extracting their trajectories. Any suitable method may be used. In certain embodiments, it may be desirable to use a low mounted camera (1-5 meters above the ground) for capturing data. For example, small vehicles like bicycles are more reliably tracked using a low mounted camera close to the intersection. In this case, problems of occlusion of tracked vehicles occur, and therefore the tracking method which is used should be robust enough to handle problems of temporary occlusion (full or partial) of tracked vehicles, and to automatically re-identify the tracked vehicle once the vehicle at least partially re-appears.
One such robust tracking method will now be detailed. This method involves from the outset searching and detecting a large number (e.g. 10<n<40) of Shi-Tomasi corners per vehicle around the center of the vehicle, and tracking these corners using a Lucas-Kanade optical flow method. If a large enough number of corners per vehicle are initially searched and detected then in case of partial occlusion in subsequent frames, at least some of the corners should still remain visible. Assuming that to be the case, the occluded vehicle can then be identified as the same vehicle as a prior tracked vehicle (i.e. in previous frames before the occlusion occurred) by matching the visible corners of the occluded vehicle (e.g. using the spatial position of the corners relative to one another) to a subset of corners of a prior tracked vehicle. Once the visible corners are matched to a specific vehicle, the location of the center point of the vehicle, even if occluded, can be determined by extrapolation based on the matched corners.
In case of full occlusion, no corners will be visible in some frames. However, assuming at least some of the corners re-appear in subsequent frames (due to movement of the occluding object, or the vehicle behind the object, or both) the object can be re-mapped if the object detection module detects the presence of the vehicle (e.g. the classifier detects the same class of vehicle at the same or near location as the occluded object within a threshold time period). In certain embodiments, a fully occluded vehicle is re-mapped to a previously acquired trajectory only in case of short duration occlusions, e.g. 0-2 seconds, in order to minimize the risk of re-mapping to the wrong trajectory (i.e. a trajectory acquired for a different vehicle). In case of full occlusions of longer durations (e.g. more than 2 seconds), tracking can be terminated at the point of the occlusion, and the acquired trajectory up to the point of the occlusion can be added to the database or, alternatively, it can be discarded.
In certain embodiments, the optical flow of corners can be used to detect incorrect tracking caused by occlusion, since it is expected that in such a case the optical flow of the corners will not be uniform and will have high standard deviation. Conversely, in the case of correct tracking with no occlusion, it is expected that all the corners of an object will have exhibit similar optical flow and thus the standard deviation of optical flow of the corners will be low. The standard deviation can also be used to identify occlusions. Upon re-identification, the object's current trajectory can be mapped to its previous trajectory which was extracted prior to the occlusion. In certain embodiments, occlusion detection may only be required at certain points at the intersection which are known in advance and can be input to the system. For example, the area before an intersection where vehicles are required to stop and queue for a traffic signal is one such typical area where there could be occlusion.
During tracking (204), a vehicle's trajectory t is extracted from the image data and recorded as an ordered set of data points d0 . . . n. Typically, a data point is collected for each frame in which the vehicle is tracked, and tracking is normally done in each frame in which the vehicle is visible (at least partially) and detectable by the tracking module. In certain embodiments, each data point d records spatial and kinematic data associated with the vehicle in a given frame. Spatial data can include, e.g. the vehicle's 2D position in the image domain (e.g. the x,y pixel coordinates of the tracked center of vehicle), the vehicle's direction in the image domain (e.g. angle with respect to a predetermined axis). In certain embodiments, a vehicle's kinematic data can include, e.g. speed (e.g. in pixels per frame), acceleration, etc. As such, the trajectory t for a vehicle v can be described as a vector of k dimensions (i.e. elements), such that
t
v
={d
0
, . . . , d
k
}, d
i={posi, spdi, acci, θi}
where
posi is the position of v in the i-th frame, e.g. the x,y coordinates of its center point;
spdi is the speed of v in the i-th frame (e.g. calculating in pixels/second as can be derived using the pos data taken from one or more previous frames, etc.);
acci is the acceleration of v in the i-th frame (e.g. calculated in pixels/(second)2 as can be derived from the spd data taken from one or more previous frames; and
By way of non-limiting example,
t
bicycle={((671.34, 277.47), 0.15, 0.02, 137.05),
((671.51, 277.60), 0.16, 0.02, 136.12),
((671.94, 277.73), 0.24, 0.03, 125.71),
((671.14, 277.39), 0.12, 0.1, 141.95)}
Processing (210) extracted trajectories to generate a plurality of reference trajectories will now be further detailed. It should be noted that the reference trajectories for each class are processed separately, though the sets of reference trajectories for different classes may be processed in parallel. First, a data removal process is applied to the extracted trajectories to remove trajectories which do not contribute a sufficient amount of data to be useable in the model. For example, short trajectories (e.g. having length L less than 400 pixels based on a frame resolution of 1920×1080 pixels), and trajectories having fewer than a certain minimum number of data points (e.g. <50) typically do not convey meaningful data and can be discarded. In certain embodiments, the data removal process may alternatively be applied during the tracking (204), in which case only trajectories meeting a minimum length and minimum number of data points are stored. Next, the remaining trajectories which are of different dimensions (i.e. different number of data points) are converted to trajectories having equal dimension data (i.e. same number of data points). Methods of converting variable dimension trajectories to equal dimension trajectories are known in the art. See, e.g. Morris and Trivedi. Briefly, a dimension k is arbitrarily chosen. The inventors have found that k=100 works well.
Next, each trajectory is sampled to extract k equidistant data points along the length L (in pixels) of the trajectory. The length of the trajectory can be calculated as the sum of the pixel distances (horizontal and vertical) between each pair of consecutive data points. First, a step size S is calculated as S=L/k. Next, the trajectory is traversed and data points d at least S pixels apart from one another are selected, beginning with the first data point d0, until the end of the trajectory is reached. After sampling, the converted trajectory is guaranteed to have k or fewer data points. In the case that the converted trajectory has fewer than k data points, the converted trajectory is resampled and additional data points are added until the number of data points equals k. The additional data points can be derived data points, e.g. derived using linear interpolation, or they can be actual data points which were removed during the sampling process. The converted trajectories are the reference trajectories r.
Clustering (212) a plurality of reference trajectories r associated with a class, thereby learning the paths available to vehicles of the class, will now be more fully detailed. Although a clustering (212) process is detailed herein to learn the available paths, other suitable path learning algorithms as known in the art can also be used. See, e.g. Morris and Trivedi in which a number of path learning techniques are detailed. In certain embodiments, k-means clustering can be used to cluster the plurality of reference trajectories associated with a class into k clusters. While k can be any value, the inventors have found that 5≦k≦20 works well. In addition, since each cluster represents a given path through the intersection, prior knowledge of the number of possible paths through the intersection that are available to be taken by vehicles of the given class can be used to set the value for k.
In certain embodiments, the reference trajectories r are clustered according to position data pos. Next, the clusters are grouped in agglomerative hierarchical fashion according to direction at end point (θk) and Euclidean distance between cluster centers with respect to the first data point, middle data point, and last data point (the cluster centers being a trajectory (either actual or derived) which is representative of the cluster). That is, for each cluster center, the shortest distances of its first data point, middle data point, and end data point from any point on each other cluster is calculated. This computation is done for all the clusters of the class. Then for each cluster its end direction is calculated. End direction can be calculated as angle between the vertical axis of the image plane and the imaginary line that would be formed by connecting the nth data point and a prior data point, such as the (0.8 n)th data point of the cluster center, where n is the number of data points on the cluster centre. Then, for any two clusters, if the absolute value of the difference between their respective end directions is less than a threshold, e.g. 30 degrees, then 3 representative distances are calculated for this cluster pair. These representative distances are shortest distance with respect to first data point, middle data point, and end data point. In other words, if the shortest distance between the first data point on the center of cluster 1 to any point on the center of cluster 2 is 50, and the shortest distance between the first data point on the center of cluster 2 to any point on the center of cluster 1 is 20, then the representative distance with respect to the first point for cluster pair 1-2 will be 20. If all three representative distances are less than a threshold, e.g. 30, and, as previously detailed the difference between the end direction of cluster centers is less than a threshold, e.g. 30 degrees, then these two clusters are grouped together. Similar calculations are done for all clusters, and all clusters which fulfil the above-mentioned distance and direction metric are grouped together. Each resulting cluster corresponds to a path available to be taken by a vehicle in the associated class. The hierarchical grouping of clusters represents path divergence. By way of non-limiting example,
In certain embodiments, clustering (212) can alternatively be performed incrementally, that is on each reference trajectory as it is generated and prior to all references trajectories having been generated.
Identifying (214) pairs of conflicting paths through the intersection and recording conflict data in a data structure will now be more fully detailed. As detailed above, a pair of paths p, q conflict when at least one reference trajectory r1 associated with path p conflicts with at least one reference trajectory r2 associated with path q. A conflict exists between two reference trajectories when the minimum distance between the trajectories is lower than a threshold (e.g. 10 pixels). The minimum distance is the smallest distance between a point on the first reference trajectory and a point on the second reference trajectory. The minimum distance between two trajectories r1 and r2 can be determined by calculating the pixel distance from each point d(1−n) in r1 to each point d(1−m) in r2 for a total of n×m distances, and taking the minimum distance.
By way of non-limiting example,
Referring now to
Processing unit (14) obtains (900) (e.g. from memory (16)) an intersection model comprising, for the intersection, a plurality of reference trajectories associated with different vehicle classes out of a predefined set of classes, the plurality of reference trajectories associated with a given class grouped into one or paths available to be taken by vehicles of the given class, as detailed above with reference to
Processing unit (14) obtains (902) (e.g. from camera (12)) image data informative of vehicles at the intersection. The process of obtaining image data which was detailed above with reference to
Processing unit (14), e.g. using classifier (26), classifies (904) a first vehicle v as belonging to a first class C(v), and a second vehicle w as belonging to a second class C(w), the first and second classes being part of the predefined set of classes used in the obtained intersection model. In certain embodiments, as detailed above, the predefined classes include “car” and “bicycle”, and one of v and w is a car and the other is a bicycle. In certain embodiments, both v and w can also belong to the same class. The process of detecting and classifying vehicles which was detailed above with reference to
Processing unit (14), e.g. using tracking module (28), tracks (906) each of v and w in the image data to extract each vehicle's (at least partial) trajectory, thereby extracting a first trajectory tv associated with v, and a second trajectory tw associated with w. The process of tracking a vehicle to extract its trajectory which was detailed above with reference to
Concurrently with tracking (906), processing unit (14) (e.g. using prediction engine (31)) selects (908), in accordance with the tv, a first set R(v) of best matching reference trajectories rv matching tv from amongst the plurality of reference trajectories associated with class C(v) comprised in the intersection model, and a second set R(w) of best matching reference trajectories rw matching tw from amongst the plurality of reference trajectories associated with class C(w) in the intersection model.
The selected reference trajectories rv in R(v) are each associated with a different path p available to be taken by v, and each reference trajectory rw in R(w) is associated with a different path q available to be taken by w. Each reference trajectory rv in R(v) is further associated with a matching cost indicative of a match between rv and tv, and each reference trajectory rw in R(w) is further associated with a matching cost indicative of a match between rw and tw. The selected reference trajectories rv in R(v) and rw in R(w) are selected on the basis of having been assigned the lowest matching cost as amongst all other reference trajectories associated with same class and the same path, as further detailed below with reference to
Using the intersection model, processing unit (14) (e.g. prediction engine (31)) identifies (910) one or more pairs of conflicting reference trajectories, a pair constituted by a first reference trajectory rv from the first set R(v) and associated with a path p predicted for v, and a second reference trajectory rw from the second set R(w) and associated with a path q predicted for w and conflicting with the first reference trajectory. Using the intersection model, processing unit (14) also identifies conflicting data points associated with each pair of conflicting reference trajectories. Examples of conflicting data points were detailed above with reference to
It should be noted that if no pairs of conflicting reference trajectories are identified, there is unlikely to be a collision between v and w, as none of the paths p predicted for v and paths q predicted for w conflict. Therefore, nothing further needs to be done, at least insofar as vehicles v and w are concerned (since it is contemplated that the CPS can concurrently track and perform collision prediction analysis on multiple vehicles and/or pairs of vehicles). On the other hand, the existence of one or more pairs of conflicting reference trajectories rv, rw is indicative of one or more pairs of conflicting paths p, q predicted for v and w, respectively, and therefore there exists a likelihood of a collision.
For each pair of conflicting reference trajectories rv, rw, processing unit (14) (e.g. prediction engine (31)) generates data (912) indicative of a likelihood of collision between v and w, as will be detailed below with reference to
Referring now to
By way of non-limiting example,
In certain embodiments, the generated data can include data indicative of the likelihood of v and w taking conflicting paths p and q, respectively, where p is the path associated with reference trajectory rv and q is the path associated with reference trajectory rw, as will further be detailed below.
Finally, processing unit (14) (e.g. warning module (32)) generates (914) a warning when the data indicative of a likelihood of collision meets a predetermined criterion, as will be further detailed below. In certain embodiments, the generated warning can be displayed (e.g. using I/O interface (18)) on a display screen mounted at the intersection and visible to at least one of v's operator and w's operator. In certain embodiments, the generated warning can be wirelessly transmitted (e.g. using communication interface (24)) to one or both of v and w for display in the vehicle using the vehicle's on-board electronics system, where the vehicle is operable to receive incoming messages from computer systems external to the vehicle (e.g. vehicles operable to receive vehicle-to-vehicle communications (V2V/V2X) messages).
In certain embodiments, the generated warning can be one of a high severity warning and a low severity warning. A high severity warning can be generated when a first predetermined criterion is met, and a low severity warning can be generated when a second predetermined criterion is met. In certain embodiments, the high severity warning can include red, flashing text warning of an imminent collision. In certain embodiments, the low severity warning can include steady, amber text warning of a possible pending collision.
By way of non-limiting example, the first predetermined criterion (for generating a high severity warning) can be met when, e.g.:
1) The likelihood of v and w taking conflicting paths is greater than 0.55 (55%); and
2) At least one of v or w will reach the point of collision in the next 3 seconds or less; and
3) v and w will reach the point of collision less than 2 seconds apart from one another.
The second predetermined criterion (for generating a low severity warning) can be met one, e.g. two of three (but not all three) of the first predetermined criterion above (for generating a high severity warning). By way of non-limiting example, the second predetermined criterion (for generating a low severity warning) can be met when, e.g.:
1) The likelihood of v and w taking a pair of conflicting paths is greater than the likelihood of v and w taking another (non-conflicting) pair of paths; and
2) At least one of v or w will reach the point of collision in the next 3 seconds or less; and
3) v and w will reach the point of collision more than 2 seconds but less than 5 seconds apart from one another;
By way of a second non-limiting example, the second predetermined criterion (for generating a low severity warning) can be met when, e.g.:
1) The likelihood of v and w taking conflicting paths is less than the likelihood of v and w taking another (non-conflicting) pair of paths; and
2) At least one of v or w will reach the point of collision in the next 3 seconds or less; and 3) v and w will reach the point of collision less than 2 seconds apart from one another.
Referring now to
Returning now to
As detailed above, a matching cost is assigned to each reference trajectory r associated with a predicted path p in accordance with r's dissimilarity to t. The matching cost assigned to a given reference trajectory r can be calculated as the sum of the pair-wise distances between the last of n data points of r and the last n data points of t, where the n data points on t represents a portion of the vehicle's current trajectory before its current position, and the n data points on r represent a corresponding portion of r before the data point on r having the least two-dimensional distance to the last data point on t. The trajectory portions used to calculate the matching cost should have the same length (i.e. same number n of data points), for example 1-20 data points. If more than one data point on each trajectory is compared, the pair-wise distance of each pair of corresponding data points can be multiplied by Gaussian weights such that the calculated matching cost is biased in favour of data points closest to the vehicle's current position (since vehicle behaviour at its current position is a better predictor of future positions than its behaviour at previous positions). Each pair-wise distance can be calculated as the sum of distances of each element position, direction, speed and acceleration.
Referring now to
In certain embodiments, the probability of v and w taking conflicting paths p, q can be determined as follows. First, a probability distribution is generated indicative of a probability of vehicles v and w vehicles taking a given pair of paths p, q using the matching cost calculated for the reference trajectories rv, rw associated with p, q, respectively. An example will now be provided. Suppose that for v there are two predicted paths p1 and p2 and for w there are two predicted paths q1 and q2. Suppose further that p1 and q1 are conflicting paths (at least one reference trajectory in p1 conflicts with at least one reference trajectory in q1). Suppose further that the matching cost for the selected reference trajectory associated with each path for each vehicle is provided in the following table:
Since a lower matching cost corresponds to a closer match, the inverse of the matching cost can be used as a similarity score indicative of the likelihood that the vehicle will take a given path. The similarity score can be used to estimate the probability P(x) of a vehicle taking a given path out of all predicted paths is provided in the following table:
The probability distribution of v and w taking a given pair of paths is provided in the table below:
As the above table illustrates, the likelihood of v and w taking conflicting paths p1 and q1, respectively, is greater than the likelihood of v and w taking a different pair of paths. As the table further illustrates, the likelihood of v and w taking p1 and q1, respectively, is greater than 50%.
It will be appreciated by one skilled in the art that many different types of warnings can be generated, and under a variety of conditions. The warning type and conditions can be implementation dependent.
It is noted that the teachings of the presently disclosed subject matter are not bound by the collision prediction system described with reference to
It is noted that the teachings of the presently disclosed subject matter are not bound by the flow charts illustrated in
It is to be understood that the invention is not limited in its application to the details set forth in the description contained herein or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Hence, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for designing other structures, methods, and systems for carrying out the several purposes of the presently disclosed subject matter.
It will also be understood that the system according to the invention may be, at least partly, implemented on a suitably programmed computer. Likewise, the invention contemplates a computer program being readable by a computer for executing the method of the invention. The invention further contemplates a non-transitory computer-readable memory tangibly embodying a program of instructions executable by the computer for executing the method of the invention.
Those skilled in the art will readily appreciate that various modifications and changes can be applied to the embodiments of the invention as hereinbefore described without departing from its scope, defined in and by the appended claims.
