Patent Application
20230394680
Document US 2014 177946 A discloses an apparatus and a method for detecting a person from an input video image with high reliability by using gradient-based feature vectors and a neural network. The human detection apparatus includes an image unit for modelling a background image from an input image. A moving object area setting unit sets a moving object area, in which motion is present by obtaining a difference between the input image and the background image. A human region detection unit extracts gradient-based feature vectors for a whole body and an upper body from the moving object area, and detects a human region in which a person is present by using the gradient-based feature vectors for the whole body and the upper body as input of a neural network classifier. A decision unit decides whether an object in the detected human region is a person or a non-person.
Document CN 104166861 A discloses a method for detection of pedestrians. The pedestrian detection method comprises the following steps: A pedestrian positive sample set and a pedestrian negative sample set needed for training a convolutional neural network are prepared. The sample sets are preprocessed and normalized to conform to a unified standard, and a data file is generated. The structure of the convolutional neural network is designed, training is carried out, and a weight connection matrix is obtained during convergence of the network. A self-adaptive background modelling is carried out on videos, information of moving objects in each frame is obtained, coarse selection is carried out on detected moving object regions at first, the regions with height to width ratios unsatisfying requirements are excluded, and candidate regions are generated. Each candidate region is input into the convolutional neural network, and whether pedestrians exist is judged.
Document US 2019 005361 AA discloses a technology for detecting and identifying objects in digital images and in particular for detecting, identifying and/or tracking of moving objects in video images using an artificial intelligence neural network configured for deep learning. In one aspect a method comprises capturing a video input of a scene comprising one or more candidate moving objects using a video image capturing device, wherein the video input comprises at least two temporally spaced images captured of the scene. The method additionally includes transforming the video input into one or more image pattern layers, wherein each of the image pattern layers comprises a pattern representing one of the candidate moving objects. The method additionally includes determining a probability of match between each of the image pattern layers and a stored image in a big data library. The method additionally includes adding one or more image pattern layers having the probability of match that exceeds a predetermined level automatically, and outputting the probability of match to a user.
Document CN 108492319 A suggests a moving object detection method based on a deep full-convolutional neural network. The method comprises the implementation steps: extracting a background image of a video scene; obtaining a multichannel video frame sequence; constructing a training sample set and a testing sample set; and carrying out the normalization of the two sample sets; constructing a deep full-convolution neural network model; carrying out the training of the deep neural network model; carrying out the prediction of the testing sample set through the trained deep full-convolution neural network model; obtaining a moving target detection result.
It is the object of the present invention to provide a method, a computer program product, a computer-readable storage medium, as well as an assistance system, by which single moving objects in the surroundings of a motor vehicle may be detected in an improved way.
This object is achieved by a method, a computer program product, a computer-readable storage medium, as well as by an assistance system according to the independent patent claims. Advantageous embodiments are indicated in the subclaims.
One aspect of the invention relates to a method for determining a motion model of a moving object in the surroundings of a motor vehicle by an assistance system of the motor vehicle. A capturing of at least one image of the surroundings with the moving object is performed by a capturing device of the assistance system. The at least one image is encoded by a feature extraction module of a neural network of an electronic computing device of the assistance system. The at least one encoded image is decoded by an object segmentation module of the neural network and a first loss function is generated by the object segmentation module. A decoding of the at least one encoded image is performed by a bounding box estimation module of the neural network and a generating of a second loss function is performed by the bounding box estimation module. The second loss function is decoded depending on the decoding of the at least one image by a motion decoding module of the neural network and a third loss function is generated by the motion decoding module. The motion model is determined depending on at least the first loss function and the third loss function by the neural network.
Thereby it is facilitated that in particular single objects can be detected in an improved way. In particular single moving objects, which are located close to each other, can be detected in an improved way. Thereby a more robust and more accurate motion segmentation may be performed.
In other words a neural network is proposed which in particular may also be referred to as convolutional neural network extracting instances of moving objects and models the respective dynamic motions individually. In order to now provide a more robust design of the neural network, prior information is incorporated into the neural network as “soft constraints”.
According to an advantageous embodiment by the bounding box estimation module a three-dimensional bounding box is generated and depending on the three-dimensional bounding box the second loss function is generated. The bounding box may in particular also be referred to as box. In other words a 3D box may be generated by the bounding box estimation module. In particular in addition to this 3D box an orientation of this 3D box can be generated. A 3D box in particular describes a reliable representation of static motor vehicles and moving pedestrians.
It has further turned out to be advantageous if a two-dimensional bounding box is generated by the bounding box estimation module and depending on the two-dimensional bounding box a fourth loss function is generated. In particular the 2D box as well as a confidence value in the image coordinates can be generated. The 2D boxes are optimized by standard loss functions for bounding boxes.
Further, it has turned out if the fourth loss function is transferred to the object segmentation module and the first loss function is generated depending on the fourth loss function. In particular the prediction of the 2D box may be trained by the combination of both the motion as well as the appearance. The 2D boxes are then fused with further information and combined in an adaptive fusion decoder in order to perform the object segmentation. This is optimized in particular by the first loss function. The first loss function is based on the semantic segmentation of pixel-by-pixel cross entropy loss by using the ground truth of the instance-based motion segmentation, in which each moving object is annotated with another value. The adaptive object segmentation module therein provides the robustness if for example one of these inputs is missing since in particular the output of the object segmentation module for the object detection is optimized.
In a further advantageous embodiment the at least one image is analyzed by a spatial transformation module of the neural network and depending on the analyzed image at least the second loss function is generated by the bounding box estimation module. The spatial transformation module may also be referred to as spatial transformer module. In particular thereby a scene geometry of the surroundings can be included, wherein a flat grid may represent the surface of a road and the spatial transformation module is trained in such a way that all information of a camera is linked to form a uniform coordinate system relative to the flat grid. Same is in particular considered by ground truths for the flat grid and the mapping of annotated objects in the three-dimensional space on the basis of extrinsic information and deep information. In particular it may further be envisaged that, even though the assumption of a flat road in many cases already works, also inclined roads may be considered within the spatial transformation module. The flat grid in this connection is subdivided into sub-grids and each grid element has a configurable inclination for an elevation, which angle as output can compensate for non-flat roads.
It is equally advantageous if for generating the second loss function the third loss function is back-propagated from the motion decoding module to the bounding box estimation module. In other words, the motion decoding module as decoder has a recurrent node in order to improve and temporally smooth the estimations of the 3D box and previous estimations of the motion model.
It is further advantageous if a first image is captured at a first point in time and a second image at a second point in time that is later than the first point in time and the first image is encoded by a first feature extraction element of the feature extraction module and the second image is encoded by a second feature extraction element of the feature extraction module and the motion model is determined depending on the first encoded image and the second encoded image. In particular thus a “two-stream Siamese encoder” for consecutive images of a video sequence image may be provided. This encoder has identical weights for the two images so that these can be effectively processed in a rolling buffer mode so that only this encoder is operated in a steady state for one output. This setup further allows the proposed algorithm to be integrated into a multi-task shared encoding system. For instance for the implementation of the encoder Resnet 18 and Resnet 50 may be used.
Further it has turned out to be advantageous if by a geometric auxiliary decoding module of the neural network a sixth loss function with geometric constraints for the object is generated and additionally depending on the sixth loss function the motion model is determined. In particular thus specific geometric restrictions or constraints of the neural network may be predetermined, under which conditions same generates the motion model. In particular these geometric constraints may for instance be determined on the basis of multi-view geometries of cameras, scene priorities based on the real geometry of road scenes, motion priorities based on the motion behavior of vehicles and pedestrians and the temporal consistency of the motion estimation.
In a further advantageous embodiment by an optical flow element of the geometric auxiliary decoding module an optical flow in the at least one image is determined and by a geometric constraint element of the geometric auxiliary decoding module the geometric constraint is determined depending on the determined optical flow. In particular the optical flow, in particular the dense optical flow, may detect a motion per pixel in the image. Thereby it is facilitated that the encoder learns motion-based features better and does not overfit on appearance cues as the typical dataset mainly contains vehicles and pedestrians as moving objects. Further, the optical flow allows incorporating the multi-view geometry of the cameras. The geometric decoder determines an optical flow and a geometric loss as sixth loss function in order to be able to incorporate epipolar constraints, positive depth/height as constraint and parallel motion constraint.
It is further advantageous if for generating the motion model a geometric mean is formed by the electronic computing device from at least the first loss function and at least the third loss function. In particular it may be envisaged that for generating the motion model the geometric mean may be formed from the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, and the sixth loss function. The explained field tests (ground truth) may possibly not generate all loss functions simultaneously. In this case the loss functions are marginalized and learned separately using asynchronous back-propagation. Further, a self-supervised learning mechanism may be used, wherein the 3D box along with the motion model of the corresponding object may be re-projected to obtain a coarse two-dimensional segment of the image which then in turn may be matched with the observed object. Since this is not a precise matching, a regulator is used to make use of corresponding tolerances. The self-supervised learning allows the reduction of large data amounts.
It is further advantageous if for determining the motion model of the moving object by means of the motion decoding module six degrees of freedom of the object are determined. In particular these degrees of freedom may comprise the directions dx, dy, dz, as well as the roll angle, the pitch angle, and the yaw angle. These six degrees of freedom are determined for each object or for every property of the moving object. The motion decoding module therein uses the output of the object segmentation module and the 3D box in order to generate an independent motion model for each moving object. The prior information relating to the moving object are encoded. The canonical three-dimensional motion of other objects is in particular either parallel to the motor vehicle or in the same direction, for instance on adjacent lanes or perpendicular to the motor vehicle. Moreover, also further motions may be learned, for instance if the motor vehicle moves itself. In field tests the parallel and perpendicular motions are separately generated and a generic motion model is generated. Then in particular the third loss function is generated based on a six-dimensional vector by field tests of a three-dimensional motion or by the estimated motion. The motion model is generated independently for each object. In particular, however, there is a dependent relationship between the respective motion models of the different objects. It may therefore be envisaged that the motion models of the different objects are merged by a graph neural network. The graph neural network thus enables an end-to-end training for an overall model for a plurality of different moving objects.
A further aspect of the invention relates to a computer program product comprising program code means, which are stored in a computer-readable medium, in order to perform the method for determining a motion model according to the preceding aspect, if the computer program product is executed on a processor of an electronic computing device.
A yet further aspect of the invention relates to a computer-readable storage medium comprising a computer program product, in particular an electronic computing device with a computer program product, according to the preceding aspect.
A yet further aspect of the invention relates to an assistance system for a motor vehicle for determining a motion model of a moving object in the surroundings of the motor vehicle, the assistance system comprising at least one capturing device and comprising an electronic computing device, which comprises a neural network with at least one feature extraction module, one object segmentation module, one bounding box estimation module, and one motion decoding module, wherein the assistance system is configured for performing a method according to the preceding aspect. In particular the method is performed by the assistance system.
A yet further aspect of the invention relates to a motor vehicle comprising an assistance system according to the preceding aspect. The motor vehicle is in particular configured as passenger car. Further, the motor vehicle is configured to be in particular at least partially autonomous, in particular fully autonomous. The assistance system may for instance be employed for the autonomous operation or for an autonomous parking maneuver.
Advantageous embodiments of the method are to be regarded as advantageous embodiments of the computer program product, the computer-readable storage medium, the assistance system, as well as the motor vehicle. The assistance system as well as the motor vehicle in this connection comprise means, which facilitate a performing of the method or an advantageous embodiment thereof.
Further features of the invention are apparent from the claims, the figures and the description of figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by the separated feature combinations from the explained implementations. Implementations and feature combinations are also to be considered as disclosed, which thus do not comprise all of the features of an originally formulated independent claim. Moreover, implementations and feature combinations are to be considered as disclosed, in particular by the implementations set out above, which extend beyond or deviate from the feature combinations set out in the back-references of the claims.
The invention now is explained in further detail by reference to preferred embodiments as well as by reference to the enclosed drawings.
These show in:
In the figures identical and functionally identical elements are equipped with the same reference signs.
In the method for determining the motion model 3 of the moving object 4 in the surroundings 5 of the motor vehicle 1 by the assistance system 2 at least one capturing of an image 20, 21 of the surroundings 5 with the moving object 4 is performed by means of the capturing device 6 of the assistance system 2. An encoding of the at least one image 21 is performed by the feature extraction module 9 of the neural network 8 of the electronic computing device 7 of the assistance system 2. The at least one encoded image 21 is decoded by the object segmentation module 10 of the neural network 8 and a generating of a first loss function 22 is performed by the object segmentation module 10.
The at least one encoded image 20, 21 is decoded by the bounding box estimation module 11 of the neural network 8 and a generating of a second loss function 23 is performed by the bounding box estimation module 11. The second loss function 23 is encoded depending on the decoding of the at least one image 20, 21 by the motion decoding module 12 of the neural network 8 and a generating of a third loss function 24 is performed by the motion decoding module 12. The motion model 3 is generated by the neural network 18 depending on at least the first loss function 22 and the third loss function 24.
In particular
Further, it is in particular shown that the at least one image 20, 21 is analyzed by the spatial transformation module 16 of the neural network 18 and depending on the analyzed image 20, 21 at least the second loss function 23 is generated by the bounding box estimation module 11.
Further,
Moreover it may be envisaged that at least a first image 20 is captured at a first point in time t1 and a second image 21 at a second point in time t2 that is later than the first point in time t1 and the first image 20 is encoded by a first feature extraction element 28 of the feature extraction module 9 and the second image 21 is encoded by a second feature extraction element 29 of the feature extraction module 9 and the motion model 3 is determined depending on the first encoded image 20 and the second encoded image 21. In particular it is further shown that by the geometric auxiliary decoding module 17 of the neural network 8 a sixth loss function 30 with geometric constraints for the object 4 is generated and additionally the motion model 3 is determined depending on the sixth loss function 30. In particular by the optical flow element 18 of the geometric auxiliary decoding module 17 an optical flow in the at least one image 20, 21 may be determined and by the geometric constraint element 19 of the geometric auxiliary decoding module 17 the geometric constraint may be determined depending on the determined optical flow.
The feature extraction module 9 thus is used as “Siamese encoder” for two consecutive images 20, 21 of a video stream. The Siamese encoder uses identical weights for the two images 20, 21 so that these effectively may run in a kind of rolling buffer so that only the encoder in the steady state is used for one output. This setup enables the proposed algorithm also to be integrated into a common multi-task shared encoder system with other tasks.
The motion segmentation module 15 is a binary segmentation decoder optimized for the fifth loss function 26. This decoder is purely optimized for the task of the motion segmentation. The ground truth annotation is based on a two-class segmentation, namely moved and static pixels.
The bounding box estimation module 11 is in particular configured as 2D/3D box decoder and outputs 2D boxes and a confidence value in image coordinates and 3D boxes in world coordinates together with the orientation. 2D boxes are optimized by using the standard bounding box loss function. Further the spatial transformation module 16 is used to incorporate a scene geometry, in which a flat grid may represent the road surface, and the spatial transformer learns to align all cameras with a uniform coordinate system relative to the flat grid. This is taken into consideration by field tests of the flat grid and the mapping of annotated objects in 3D on the basis of extrinsic information and depth estimation. Also inclined roads may be present, which equally may be integrated into the spatial transformation module 16. The flat grid is subdivided into sub-grids and each grid element has a configurable inclination, which may be output to compensate for non-flat roads.
For the object segmentation module 10 the 2D box prediction is trained in such a way that it is a combination of motion and appearance. The 2D boxes are merged with the motion segmentation output of the motion segmentation module 15 by using an adaptive fusion decoder. This is optimized by the first loss function 22. The first loss function 22 is based on a semantic segmentation with pixel-by-pixel cross entropy loss by using field tests of an instance-based motion segmentation, in which each moving object 4 is annotated with a different value. The adaptive fusion facilitates a robustness, if one of the inputs is missing, as the fusion output is for instance optimized for the detection.
The motion decoding module 12 is a module, in which the 3D motion (six degrees of freedom dx, dy, dz, yaw angle, pitch angle, and roll angle) is estimated for each case of a moving object 4. This decoder makes use of the output of the object segmentation module 11, which is represented in particular by the arrow 31, and the output of the 3D box in order to generate an independent motion model 3 for each moving object 4. This decoder also has a back-propagation in order to improve and temporally smooth the estimations of the 3D box. Prior information as to the motion model 3 are used, such as for instance a canonical 3D motion of other objects 4, which are either parallel to the motor vehicle 1 on the same or adjacent lanes or perpendicular thereto. Even though there are also other motions, such as for instance a rotation/turning of the motor vehicle 1, it is advantageous to specialize and learn these motions separately. By field tests the parallel and the perpendicular motions are separated and a generic motion model 3 is generated also for the other cases. The motion model 3 is modelled independently for each object 4. However, there is a dependence of the motion models 3. The motion models 3 of the individual objects 4 may therefore be merged for instance via a graph neural network. The modelling via the graph neural network facilitates an end-to-end training for the complete model.
In the geometric auxiliary decoding module 17 a dense optical flow is generated on the basis of an image-based motion per pixel. Thereby the encoder is forced to learn motion-based features better and not to overfit on appearance cues/since the typical dataset mainly contains vehicles and pedestrians as moving objects 4. Moreover, the optical flow allows the incorporation of geometric constraints for several views. The proposed geometric decoder computes the dense optical flow and a geometric loss, in particular the sixth loss function 30, is determined in order to integrate epipolar constraints, positive depth/height constraint, and parallel motion constraint.
The overall loss function is in particular a geometric product of the individual loss functions 22, 23, 24, 25, 26, 30. The corresponding field tests possibly are not available for all these loss functions 22, 23, 24, 25, 26, 30 simultaneously. In this case they can be marginalized and learned separately by using an asynchronous back-propagation. Further, a self-supervised learning is proposed, in which the 3D box together with the motion model 3 of the corresponding object 4 can be newly projected in order to obtain a coarse 2D segment on the image which matches the observed object 4. Since this is not a precise matching, a regulator for the matching is used in order to allow tolerances. The self-supervised learning facilitates compensating for a lack of large data amounts.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2020 126 690.1 | Oct 2020 | DE | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/077506 | 10/6/2021 | WO |
