Patent Application
20240012126
This application claims priority to European Patent Application No. EP22184160.4, filed Jul. 11, 2022, the disclosure of which is incorporated by reference in its entirety.
In recent years, development in the field of driver assistance systems and systems for autonomous driving has advanced. The use of such systems in vehicles serves to support the driver and, for example, to detect and prevent dangerous situations. Driving assistance systems are now ubiquitous and, in some cases, even mandatory. The known driving assistance systems use incoming data such as data from sensors such as cameras, laser scanners, radar sensors, etc. and determine from this, usually using techniques of machine learning, outputs in the form of control information for controlling the vehicle or other information for display to the driver of the vehicle. However, so far not all inputs from sensors, in particular radar sensors, are usable for processing in driving assistance systems due to their peculiarities and characteristics such as randomness.
The subject disclosure therefore aims to provide improvements for existing driving assistance systems and their detection mechanisms.
The subject disclosure is directed to improve known prediction mechanisms.
In this respect, according to a first aspect, a computer-implemented method for a driving assistance system of a vehicle is provided. The method comprises determining a plurality of detection points obtained by a sensor, the detection points being associated with a plurality of segments of a region of interest for the vehicle and representing objects in the plurality of segments. The method further comprises, for each segment of the plurality of segments, encoding the plurality of detection points associated to said segment into a latent feature of said detection point, and determining a fixed feature for said segment based on the latent features of the plurality of detection points. The method further comprises encoding the fixed features of the plurality of segments to obtain a spatial correlation of static detection points among the segments, the spatial correlation of static detection points indicating a characteristic of a driving corridor for the vehicle and providing the spatial correlation of static detection points to the driving assistance system.
In some embodiments, the method further comprises receiving the plurality of detection points from the sensor of the vehicle.
In some embodiments, the method further comprises dividing the region of interest into the plurality of segments. In such embodiments, dividing the region of interest may optionally comprise dividing the region of interest into the plurality of segments in a longitudinal direction of a detection area of the sensor, each segment having a given size in the longitudinal direction.
In some embodiments, each of the plurality of detection points is represented by a feature vector having a fixed number of features obtained by the sensor and the latent feature is represented by a feature vector having a number of features, equal or greater than the fixed number of features, e.g., obtained by the sensor.
In some embodiments, the method further comprises, for each segment of the plurality of segments of the region of interest for the vehicle, determining whether a number of detection points obtained by the sensor exceeds a given maximum, and obtaining the plurality of detection points from among the number of detection points obtained by the sensor such that the plurality of detection points corresponds, in number, to the given maximum. In such embodiments, obtaining the plurality of detection points may optionally comprise one or more of selecting, based on one or more given criteria, the plurality of detection points from among the number of detection points, and performing padding with respect to the number of detection points.
In some embodiments, determining the fixed feature for said segment comprises applying, on the latent features of the plurality of detection points, a symmetric function to obtain the fixed feature for said segment. In such embodiments, the symmetric function may comprise a max pooling function.
In some embodiments, encoding each of the plurality of detection points and determining the fixed feature for said segment comprises using a segment encoding module of a machine learning network for encoding each detection point into the latent feature and applying the symmetric function to obtain the fixed feature for said segment. In such embodiments, the segment encoding module may be trained for one or more of the plurality of segments.
In some embodiments, the method further comprises concatenating the fixed features of the plurality of segments in an order of the segments.
In some embodiments, encoding the fixed features of the plurality of segments comprises using a correlation encoding module of the machine learning network. In such embodiments, the correlation encoding module may be trained for the plurality of segments and/or the spatial correlation of the static detection points.
In some embodiments, the sensor is a radar sensor of the vehicle, and the plurality of detection points comprises a plurality of reflection points corresponding to radar reflections from the objects.
In some embodiments, the method further comprises one or more of controlling the vehicle in accordance with control information, the control information determined by the driving assistance system based at least in part on the spatial correlation of static detection points, and providing visualization information relating to the characteristic of the driving corridor, the visualization information determined by the driving assistance system based at least in part on the spatial correlation of static detection points.
According to a second aspect, a processing apparatus is provided. The processing apparatus comprises a processor configured to perform the method according to the first aspect and embodiments thereof. Alternatively, or additionally, the processing apparatus may comprise a processor and a memory including computer program code. The computer program code, when executed with the processor, causes the processing apparatus to perform the method according to the first aspect and embodiments thereof. Alternatively, or additionally, the processing apparatus may comprise one or more means configured to perform the method according to the first aspect and embodiments thereof.
According to a third aspect, another processing apparatus is provided. The processing apparatus comprises a segment encoder to, for each segment of a plurality of segments of a region of interest for a vehicle, encode a plurality of detection points representing objects in said segment into a latent feature of said detection point, the plurality of detection points being obtained by a sensor, and determine a fixed feature for said segment based on the latent features of the plurality of detection points. The processing apparatus further comprises a correlation encoder to encode the fixed features of the plurality of segments to obtain a spatial correlation of static detection points among the segments, the spatial correlation of static detection points indicating a characteristic of a driving corridor for the vehicle. The processing apparatus further comprises an interface to provide the spatial correlation of static detection points to a driving assistance system.
According to a fourth aspect, a driving assistance system for a vehicle is provided. The driving assistance system comprises the processing apparatus according to the second and/or third aspect. An autonomous driving system for a vehicle may also be provided with the processing apparatus according to the second and/or third aspect.
According to a fifth aspect, a vehicle is provided with a sensor for detecting objects in a region of interest, and a driving assistance system according to the fourth aspect. Alternatively, or additionally, the vehicle may comprise the sensor for detecting objects in a region of interest and the autonomous driving system.
According to a sixth aspect, a computer program product is provided. The computer program product comprises instructions which, when executed by a computer, cause the computer to carry out the method according to the first aspect and embodiments thereof.
This Summary is intended to provide a brief overview of some of the aspects and features according to the subject disclosure. Accordingly, it will be appreciated that the above-described features are merely examples and should not be construed to narrow the scope of the subject disclosure in any way. Other features, aspects, and advantages of the subject disclosure will become apparent from the following detailed description, drawings, and claims.
A better understanding of the subject disclosure can be obtained when the following detailed description of various example embodiments is considered in conjunction with the accompanying drawings, in which:
The subject disclosure generally relates to advanced driving assistance systems and autonomous driving systems for vehicles. More specifically, the subject disclosure relates to methods, apparatuses, and systems for integrating radar detection for prediction for advanced driving assistance systems and autonomous driving systems.
Before explaining the example embodiments of the subject disclosure in more detail, certain general principles relating to the subject disclosure are briefly explained with reference to
Referring first to
A driving assistance system as used herein may have a variety of different inputs and outputs. The inputs include, for example, trajectories 110 of vehicles 100. Such trajectories 110 include a series of discrete points 120 with associated features, and lane markings 130 (shown by the boundary of a road on which the vehicle 100 is travelling). Such lane markings 130 are represented, for example, by continuous third order or polynomial functions and comprise continuous or discrete features. In addition, (static) detection points 140 from a sensor such as a radar sensor may provide useful input to the driving assistance system. Such points 140 typically have features such as position, speed, etc. However, due to the peculiarities of detection points 140, which are created by reflection of radar waves from the radar sensor on objects, it may be challenging to derive or predict information, such as a characteristic of a driving corridor of the vehicle 100, since the spatial correlation of the detection points exhibits randomness.
The randomness in number of detection points, position of detection points, etc. is illustrated in
In the scenario of a road, the detection points typically concern reflections of objects outside the possible driving corridor of a vehicle. In
In view of the above, the subject disclosure is to provide improvements for existing driving assistance systems and their detection mechanisms. In particular, the subject disclosure is aimed at providing a solution for processing detection points from radar sensors using a machine learning-based approach. Furthermore, the subject disclosure aims at an easy (plug & play) integration of the solution into the existing driver assistance systems and their detection mechanisms. In particular, the subject disclosure considers the characteristics of static radar points.
Now, the methods for a driving assistance system of a vehicle 200 according to some embodiments of the subject disclosure will be described.
Referring now to
The example method is implemented by a processing device such as the computing system described below with reference to
For the purposes of this description, vehicles are typically vehicles for land transportation, such as passenger cars or trucks that participate in public road traffic and use roads. However, it is to be understood that the subject disclosure is not limited thereto and relates, for example, to other vehicles for transport on land, such as motorbikes, as well as vehicles for water and air transport. The latter may comprise, for example, the use of the example method in an aircraft, where the example method serves to control movement or driving of the aircraft on taxiways, runways, or the apron of airfields.
The example method of
In some embodiments, the example method may include receiving the plurality of detection points from the sensor of the vehicle. The detection points represent reflection points of objects in the ROI and are obtained by the sensor by scanning a region of interest (ROI). For this purpose, the radar sensor, for example, sends electromagnetic waves (radar waves) in the direction of the ROI. The radar waves are reflected by objects (e.g., other moving or parked vehicles, guideposts, guardrails, kerbs, buildings, trees, grass, etc.) in the ROI. It should be understood here that the objects include static and/or dynamic objects in relation to the road. The radar sensor detects the reflections from the objects by receiving electromagnetic waves reflected from the objects. The reflections are received by the sensor as points and/or processed by processing devices downstream of the sensor in the form of points. These points are referred to herein as detection points.
Also disclosed is an activity referred to herein as segment encoding 320. This functionality is used to determine one or more representative static points for each of the plurality of segments into which the ROI with respect to the vehicle is divided. These points, hereinafter also referred to as static detection points, or their distribution in the ROI, typically indicate or represent markers at which a characteristic of the ROI can be determined. For example, in the case of a vehicle such as a passenger car or truck on a road, the static detection points indicate a characteristic of a driving corridor (e.g., the road on which the vehicle is moved or driven). As will be described in more detail below, the ROI is divided into a plurality of segments.
According to the example method, the segment encoding 320 is performed for each segment of the plurality of segments of the ROI. Accordingly, the segment encoding 320 may be performed sequentially for the plurality of segments. However, it is not necessary that the processing of the segments is performed in the order of the segmentation of the ROI, for example, from the segment closest to the sensor (or the vehicle) in the ROI to the segment furthest from the sensor (or the vehicle) in the ROI. Similarly, segment encoding 320 may comprise parallel processing of segments, for example in groups of two or more adjacent segments, or even parallel processing of all segments of the plurality of segments.
For each segment, segment encoding 320 encodes one or more detection points (typically a plurality of detection points) of objects within the respective segment into what is herein referred to as a latent feature (operation 324 in
In some embodiments, encoding the plurality of detection points in operation 324 comprises using a machine learning network or a module of a machine learning network (hereinafter also referred to as a segment encoding module). The machine learning network or segment encoding module of the machine learning network is a network trained with respect to the detection points in the respective segment or in several (possibly all) of the plurality of segments. Encoding in operation 324 “enriches” the detection points with additional information. In accordance with some embodiments, each detection point has a certain number of features detected or determined by the sensor or an associated processing device for a corresponding reflection from an object. Accordingly, typically all detection points obtained from the sensor have the same number of features. The features may be, for example, position, distance from the sensor, strength of reflection, etc. A detection point therefore represents a feature vector with a certain number of features. For example, the number of features may be 3. By means of encoding in operation 324, which in the field of machine learning is also referred to as feature embedding, each detection point (e.g., corresponding feature vector for the detection point with the certain number of features) is encoded into a latent feature by embedding additional features. The latent feature is also a feature vector, but—in some embodiments—with a number of features (for example 5) greater than the certain number of features of the feature vector for the detection point.
In addition, segment encoding 320 includes a determination of a new feature vector, referred to herein as a fixed feature for the segment, having a certain number of features (operation 326 in
With the determination of the fixed feature for the segment, the segment encoding 320 ends.
The example method of
In activity 330, the correlation encoding, the fixed features of the plurality of segments determined in activity 320 are encoded to obtain a spatial correlation of static detection points among the segments. Static detection points are defined here as detection points of objects that are determined to be static with respect to the segments or the ROI. In other words, static detection points represent points that are fixed and reliable for detecting the characteristics of the environment (or the ROI). In relation to the vehicle, the static detection points or the spatial correlation thereof indicate a characteristic of a driving corridor for the vehicle. The static detection points or the spatial distribution thereof “approximate” the course of the road on which the vehicle is moving or being driven, and thus an area (referred to here as the driving corridor) in which the vehicle should move or be driven. It is to be understood that the correlation encoding in activity 330 considers the order of the segments according to the segmentation of the ROI to determine the spatial correlation of the static detection points. Typically, the correlation encoding in activity 330 considers the order of the segments in the ROI from the segment closest to the sensor (or vehicle) in the ROI to the segment furthest from the sensor (or vehicle) in the ROI. Reversing the order from the furthest segment to the closest segment is also possible.
In some embodiments, similar to the encoding in activity 324, encoding in activity 326 also comprises using a machine learning network or module of a machine learning network (hereinafter also referred to as a correlation encoding module). The machine learning network or correlation encoding module of the machine learning network is a network trained with respect to the static detection points or the spatial distribution or correlation thereof. The machine learning network containing the correlation encoding module may be the network also containing the segment encoding module; the segment encoding module and the correlation encoding module may accordingly be part of the same machine learning network. The encoding in activity 326 is to determine or derive additional information based on the fixed features among the segments. For example, the encoding in activity 326 relates to a determination which fixed features may qualify as static detection points and thus allow to determine or derive the characteristic of the driving corridor for the vehicle.
With the determination of the static detection points or the spatial correlation thereof, the correlation encoding of activity 330 ends.
In the basic flow chart of
As described below in some more detail, the static detection points provided or the spatial correlation thereof may be used, for example, by the driving assistance system to control the vehicle or to provide visual information (for example). Thus, the information provided by the example method (e.g., the static detection points) may be used to determine control information for controlling the vehicle.
In some embodiments, the static detection points indicating a characteristic of a driving corridor for the vehicle, as described, may be used to determine a trajectory for the vehicle and derive control information therefrom to control or steer the vehicle along the trajectory. Likewise, control information may be determined that, for example, causes a steering intervention for the vehicle in such a way that the vehicle moves (further) on the road or in the driving corridor.
In other embodiments, the static detection points may be used to generate visual information representing the characteristics of the driving corridor. The visual information may, for example, be used to indicate to a driver of the vehicle, for example by means of a display (e.g., the fully digital instrument cluster mentioned above), the course of the road ahead of the vehicle. Likewise, the visual information may be used—independently of a vehicle or a driving assistance system—to show a person the characteristics or course of a road, for example by means of a head-up display (HUD). This can be useful, for example, for driving simulation, virtual driving training and/or testing/certification of vehicle components such as ADAS or ADS.
Some further embodiments of the example method according to
As described above, segment encoding 320 is performed for each of a plurality of segments into which the ROI is divided with respect to the vehicle. The sensor of the vehicle typically has a detection area (also referred to as sensing area) that corresponds to or includes the ROI, as assumed herein. The detection area of the sensor or the ROI is divided into the plurality of segments. The segmentation is done virtually. Typically, the segmentation of the ROI is done in a longitudinal direction of the detection area or a main detection direction of the sensor.
An example segmentation of the ROI into a plurality of segments is shown in
In some embodiments, the segmentation of the ROI into the plurality of segments is such that the segments have the same size in the longitudinal direction of the detection area or the main detection direction of the sensor, as shown in
In some embodiments, the sensor and/or a processing device coupled to the sensor may associate with the detection points detected by the sensor information relating to the segment to which the detection points belong or in which the objects causing the detection points by reflection are located. In other embodiments, the assignment of the detection points may be based on information from the sensor (e.g., coordinates of a plane with the sensor, distance to the sensor). Information about the segment to which a detection point belongs may be assigned to the detection point or provided in addition. A numbering or a unique index may be suitable for this purpose. The segment coding 320 according to the example method may receive the plurality of detection points from the sensor and (together or additionally) the information about the segments to which the detection points are associated. Segment encoding 320 may then use the segment information to determine those detection points of the plurality of detection points that are associated with a segment to be subjected to segment encoding 320. In other embodiments, only those detection points of a segment that is to be subjected to the segment encoding 320 may be provided to the example method as shown in
Furthermore, in some embodiments, a maximum number of detection points may be given for a segment for the segment encoding 320 of the example method according to
In some embodiments, obtaining the plurality of detection points corresponding to the maximum number may comprise selecting detection points from the number of detection points obtained by the sensor. The selecting may be based on one or more predetermined criteria. For example, the selection may be based on one or more features associated with the detection points. For example, detection points with higher strength of reflection (e.g., a strength of reflection above a predetermined threshold) may be selected and thus preferred compared to detection points with lower strength of reflection. In another example, a detection point representing a group of detection points may be selected from the group of detection points having a given spatial proximity. By selecting detection points, their number is reduced to the maximum number.
In other embodiments, obtaining the plurality of detection points corresponding to the maximum number may comprise performing padding with respect to the number of detection points obtained by the sensor. Examples of padding may include the so-called zero-padding (e.g., adding detection points that do not affect segment encoding 320, and do not falsify the result from segment encoding 320) or copying detection points from the number of detection points for a segment. Padding increases the number of detection points to the maximum number.
Further, in some embodiments, the example method may comprise applying a symmetric function to the latent features of the plurality of detection points in operation 326. By applying the symmetric function, the fixed feature for the segment for which the segment encoding 320 is performed is determined based on the latent features. An example symmetric function, which is typically used in the field of machine learning, comprises the so-called Max-pooling. Max-pooling is a sample-based discretization process with the objective to down-sample an input representation, reducing the dimensionality and allowing for assumptions to be made about features contained in the sub-regions binned.
Alternatively, or additionally, the example method according to
It has already been described above that the machine learning network or the segment encoding module or the correlation encoding module of the machine learning network is a trained network. In some embodiments, the segment encoding 320 uses the same segment encoding module (e.g., a segment encoding module that uses the same parameters for the segments) of the network for multiple segments. In this example, segment encoding 320 is performed using a segment encoding module associated with those multiple segments for which segment encoding 320 is to be performed. It is understood that the same segment encoding module may be used for all the plurality of segments. In other examples, a different segment encoding module (e.g., different segment encoding modules that use different parameters) of the network may be used for each segment in the segment encoding 320. In such an example, the segment encoding 320 is performed using a segment encoding module associated with the segment for which the segment encoding 320 is to be performed.
In the aforementioned examples, the segment encoding module may be trained for one or more segments of the plurality of segments (e.g., with parameters for those segments). The one or more segments may be associated with the segment encoding module as described above.
The correlation encoding module used in the correlation encoding 330 may, in some embodiments, be trained for the plurality of segments and/or the sequence, for static detection points and/or the spatial correlation/distribution thereof.
The example method described herein for a driving assistance system in a vehicle as well as its embodiments enable that, despite the existing randomness in the data (e.g., detection points of reflections from objects) from a (radar) sensor, information (referred to herein as the static detection points) is available or can be extracted or derived to support or improve the driving assistance system (or system for autonomous driving). The subject disclosure is therefore directed to the properties and characteristics contained in the data from the sensor.
A simple application of the example method described above will now be described.
For this purpose, reference is first made again to the illustration of
According to the example method, at least in some embodiments, a segmentation of the ROI is performed first, e.g., the scenario shown in
According to the example method, the determination of the static detection points or the spatial correlation or distribution thereof is performed in two phases: segment encoding 320 and correlation encoding 330. In segment encoding 320, for each segment (e.g., for each area between two adjacent dot-dashed lines 420 in
Reference is made below to
In the subject disclosure, the use of a machine learning network is mentioned. Such a network may be a convolutional neural network (CNN), but also a recurrent neural network (RNN) or fully connected (FC) network. In particular, a CNN may be used for the segment encoding module and the correlation encoding module, which may be effectively used to learn a longitudinal distribution of detection points, in particular due to the described segment-wise processing of the detection points.
The example method described above according to
The computing system 800 may be in the vehicle and includes at least one processor 810, a user interface 820, a network interface 830 and a main memory 860, that communicate with each other via a bus 850. Optionally, the computing system 800 may further include a static memory 870 and a disk-drive unit (not shown) that also communicate with each via the bus 850. A video display, an alpha-numeric input device and a cursor control device may be provided as examples of the user interface 820.
Furthermore, the computing system 800 may also comprise a sensor interface 840 to communicate with a sensor of the vehicle. Alternatively, the computing system 800 may communicate with the sensor via the network interface 830. The sensor obtains the plurality of detection points (e.g., reflection points from objects) from the ROI for the vehicle. The sensor may be a radar sensor, a laser scanner, a light detection and ranging (Lidar) sensor, or the like. The sensor may be directed into any direction of interest to the vehicle such as the front direction (e.g., the driving direction). The computing system 800 may also be connected to database systems (not shown) via the network interface 830, wherein the database systems store additional data needed for providing the functionalities as described herein.
The main memory 860 may be a random-access memory (RAM) and/or any further volatile memory. The main memory 860 may store program code 880 for performing the example method described herein. The memory 860 may also store additional program data 882 required for providing the functionalities as described herein. Part of the program data 882 and/or the program code 880 may also be stored in a separate, e.g., cloud memory and executed at least in part remotely. In such an example embodiment, the memory 860 may store the segment encoding module and the correlation encoding module as described herein in a cache 890.
Computer readable storage media, which are inherently non-transitory, may include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer readable storage media may further include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technology, portable compact disc read-only memory (CD-ROM), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be read by a computer.
A computer readable storage medium should not be construed as transitory signals per se (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission media such as a waveguide, or electrical signals transmitted through a wire). Computer readable program instructions may be downloaded to a computer, another type of programmable data processing apparatus, or another device from a computer readable storage medium or to an external computer or external storage device via a network.
It should be appreciated that while particular embodiments and variations have been described herein, further modifications and alternatives will be apparent to persons skilled in the art. In particular, the examples are offered by way of illustrating the principles, and to provide a number of specific methods and arrangements for putting aspects of the subject disclosure into effect.
In certain embodiments, the functions and/or acts specified in the flow charts, sequence diagrams, and/or block diagrams may be re-ordered, processed serially, and/or processed concurrently without departing from the scope of the invention. Moreover, any of the flowcharts, sequence diagrams, and/or block diagrams may include more or fewer blocks than those illustrated consistent with embodiments of the invention.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the subject disclosure. It will be further understood that the terms “comprise” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Furthermore, to the extent that the terms “include”, “having”, “has”, “with”, “comprised of”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising”.
While a description of various embodiments has illustrated all the subject disclosure and while these embodiments have been described in considerable detail, it is not the intention to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. The subject disclosure in its broader aspects is therefore not limited to the specific details, representative apparatus and method, and illustrative examples shown and described. Accordingly, the described embodiments should be understood as being provided by way of example, for the purpose of teaching the general features and principles, but should not be understood as limiting the scope, which is as defined in the appended claims.
Unless context dictates otherwise, use herein of the word “or” may be considered use of an “inclusive or,” or a term that permits inclusion or application of one or more items that are linked by the word “or” (e.g., a phrase “A or B” may be interpreted as permitting just “A,” as permitting just “B,” or as permitting both “A” and “B”). Also, as used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. For instance, “at least one of a, b, or c” can cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c, or any other ordering of a, b, and c). Further, items represented in the accompanying figures and terms discussed herein may be indicative of one or more items or terms, and thus reference may be made interchangeably to single or plural forms of the items and terms in this written description.
| Number | Date | Country | Kind |
|---|---|---|---|
| 22184160.4 | Jul 2022 | EP | regional |
