Patent Application
20230274142
The present application claims the benefit under 35 U.S.C. § 119 of German Patent Application No. DE 10 2022 202 030.8 filed on Feb. 28, 2022, which is expressly incorporated herein by reference in its entirety.
The present invention relates to a method for training a conditional neural process for determining a position of an object from image data, and in particular to a method for training a conditional neural process for determining a position of an object from image data, with which a conditional neural process for determining a position of an object from image data with optimized performance can be trained with comparatively low resource consumption.
The term “meta-learning algorithm” is understood to mean a machine learning algorithm designed to optimize the algorithm by autonomous learning as well as drawing on experiences. Such meta-learning algorithms are in particular applied to metadata, wherein the metadata maybe, for example, properties of the corresponding learning problem, algorithm properties or patterns, which were previously derived from the data. The application of such meta-learning algorithms in particular has the advantage that the performance of the algorithm can be increased and the algorithm can be adapted quickly and flexibly to different problems and/or new categories of objects. Such meta-learning algorithms are used, for example, to determine a position and/or pose, or 6D-pose, of an object based on image data.
Meta-learning algorithms include, for example, model-agnostic meta-learning (MAML) or conditional neural processes. The aim of these algorithms is to optimize model parameters in such a way that training success can be achieved with comparatively few gradient optimizations. Conditional neural processes are in particular based on using a feed-forward neural network to calculate the training data information, to aggregate this information, and to transmit this information to another feed-forward network for inference.
However, it proves disadvantageous with such meta-learning algorithms, for example, that the training of such algorithms is comparatively complex and can lead to so-called overfitting or memorization of training data. In particular, during the training of such an algorithm, a state can occur in which only problem solutions determined from the training data are reproduced, that is, the algorithm correctly processes only the training data and does not achieve any new results when new data are input.
PCT Patent Application No. WO 2019/099305 A1 describes a method for automating the learning of several tasks by a single neural network based on meta-learning, wherein the order in which tasks are learned by the neural network can affect the performance of the network, and wherein a task-level plan can be used for learning the several tasks. The plan provides for monitoring a course of cost functions during the training, wherein compensatory weights for task losses can be adjusted in the course of the training.
An object of the present invention is to provide an improved method for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data.
The object may be achieved with a method for training a conditional neural process for determining a position of an object from image data according to the features of the present invention.
The object may also be achieved with a control device for training a conditional neural process for determining a position of an object from image data according to the features of the present invention.
According to one example embodiment of the present invention, this object may be achieved by a method for training a conditional neural process for determining a position of an object from image data, wherein the method comprises providing training data for training the conditional neural process, wherein the training data comprise labeled image data showing a particular object and labeled comparison image data regarding the particular object; and training the conditional neural process based on the provided training data, wherein the training of the conditional neural process comprises applying functional contrastive learning, and wherein the training of the conditional neural process comprises applying an end-to-end learning approach.
The term “image data” is understood to mean data that are generated by scanning or optically recording one or more surfaces by means of an optical or electronic device or an optical sensor.
The image data showing a particular object are image data that show a surface on which the particular object is placed or positioned, and were recorded for training purposes.
The comparison image data regarding the particular object furthermore are comparison or context data and in particular digital images, which likewise represent the respective particular object for comparison or as a reference.
The term “labeled data” is furthermore understood to mean already known data that have already been prepared, for example, from which features, such as the position or nature of individual objects in the corresponding image data have already been extracted or from which patterns have already been derived.
Contrastive learning furthermore consists in learning a metric space between two sample values by reducing the distance between two positive sample values while increasing the distance between two negative sample values. The term “functional contrastive learning” is in particular understood to mean an algorithm designed to reduce the distance between two corresponding representations, in particular the distance or difference between two representations relating to the same task or the same object, and to find matching representations.
The term “end-to-end learning approach” is furthermore understood to mean an approach based on input and output data of a neural network, wherein the neural network is trained on output data desired with respect to an input or corresponding input data.
The combination of functional contrastive learning and an end-to-end learning approach in particular has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks.
Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
Specified overall is thus an improved method for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data.
According to an example embodiment of the present invention, the step of training the conditional neural process based on the provided training data can in this case comprise generating first latent representations based on the labeled image data and information about the labeled image data; generating second latent representations based on the labeled comparison image data and information about the labeled comparison image data; determining, by means of the functional contrastive learning, a first cost function based on the first latent representations and the second latent representations; and training the conditional neural process based on the first cost function.
The term “latent representations” is understood to mean intermediate states of the input data or image data during the processing of the image data by the conditional neural process, wherein the latent representations usually have a smaller dimension than the original image data.
The term “information about the labeled image data or labeled comparison image data” is furthermore understood to mean information about the patterns or labels contained in the comparison image data, for example, information about the position of individual objects represented in the image data or comparison image data.
The term “cost function” or “loss” is furthermore understood to mean a loss or an error between determined output values and corresponding actual circumstances or actual measured data.
Overall, the conditional neural process can thus be trained in a simple manner with simultaneously comparatively low resource consumption, wherein the performance of the trained conditional neural process can simultaneously be optimized.
According to an example embodiment of the present invention, the step of training the conditional neural process based on the provided training data may furthermore also comprise determining, by means of the conditional neural process, a position of the particular object in the image data based on the labeled image data, the labeled comparison image data and information about the labeled comparison image data; determining a comparison position of the particular object in the labeled image data based on information about the labeled image data; determining a second cost function based on the determined position of the particular object in the image data and the comparison position of the particular object; and training the conditional neural process based on the second cost function.
The conditional neural process can also again be trained thereby in a simple manner with simultaneously comparatively low resource consumption, wherein the performance of the trained conditional neural process can simultaneously be optimized.
In one example embodiment of the present invention, the image data and the comparison image data respectively are image data showing complete images.
The term “image data showing complete images” or “higher-dimensional image data” is understood to mean image data that characterize, or represent, not only a part, for example, a two-dimensional portion of an image or individual pixels of an image, but the complete image.
In particular, the method according to the present invention can train a conditional neural process designed to process even complete images in a simple manner or to determine the position of objects from complete images in a simple manner, wherein the performance of a correspondingly trained conditional neural process can be optimized even further.
With a further example embodiment of the present invention, a method for determining a position of an object is also specified, wherein the method comprises providing image data, wherein the image data comprise target image data showing the object and labeled comparison image data regarding the object; providing a conditional neural process, trained by a method described above for training a conditional neural network for determining a position of an object from image data, for determining a position of an object from image data; and determining, by means of the provided conditional neural process for determining a position of an object from image data, the position of the object based on the provided image data.
Such a method for determining a position of an object has the advantage that it is based on an improved method for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data. In particular, the combination of functional contrastive learning and an end-to-end learning approach in the training of the conditional neural process has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks. Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
With a further example embodiment of the present invention, a method for controlling a controllable system is also specified, which comprises determining a position of an object from image data by means of a method described above for determining a position of an object, and controlling a controllable system based on the determined position of the object.
The controllable system may, for example, be a robotic system, wherein the robotic system may in turn be a gripper robot, for example. However, the system may also be, for example, a system for controlling or navigating an autonomously driving motor vehicle or a system for face recognition.
Such a method for controlling a controllable system has the advantage that it is based on an improved method for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data. In particular, the combination of functional contrastive learning and an end-to-end learning approach in the training of the conditional neural process has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks. Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
With a further example embodiment of the present invention, a control device for training a conditional neural process for determining a position of an object from image data is also specified, wherein the control device comprises a provisioning unit designed to provide training data for training the conditional neural process, wherein the training data comprise labeled image data showing a particular object and labeled comparison image data regarding the particular object; and a training unit designed to train the conditional neural process based on the provided training data, wherein the training of the conditional neural process comprises applying functional contrastive learning, and wherein the training of the conditional neural process comprises applying an end-to-end learning approach.
Specified is thus an improved control device for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data. In particular, the combination of functional contrastive learning and an end-to-end learning approach in the training of the conditional neural process has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks. Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
In this case, according to an example embodiment of the present invention, the training unit may furthermore comprise a first generation unit designed to generate first latent representations based on the labeled image data and information about the labeled image data; a second generation unit designed to generate second latent representations based on the labeled comparison image data and information about the labeled comparison image data; and a first determination unit designed to determine, by means of the functional contrastive learning, a first cost function based on the first latent representations and the second latent representations, wherein the training unit may be designed to train the conditional neural process based on the first cost function. Overall, the training unit can thus be designed in such a way that the conditional neural process can be trained in a simple manner with simultaneously comparatively low resource consumption, wherein the performance of the trained conditional neural process can simultaneously be optimized.
Moreover, according to an example embodiment of the present invention, the training unit may furthermore comprise a second determination unit designed to determine, by means of the conditional neural process, a position of the particular object in the image data based on the labeled image data, the labeled comparison image data and information about the labeled comparison image data; a third determination unit designed to determine a comparison position of the particular object in the labeled image data based on information about the labeled image data; and a fourth determination unit designed to determine a second cost function based on the determined position of the particular object in the image data and the comparison position of the particular object, wherein the training unit may be designed to train the conditional neural process based on the second cost function. The conditional neural process can also again be trained thereby in a simple manner with simultaneously comparatively low resource consumption, wherein the performance of the trained conditional neural process can simultaneously be optimized.
In one example embodiment of the present invention, the image data and the comparison image data respectively are image data showing complete images. In particular, the control device according to the present invention can train a conditional neural process designed to process even complete images in a simple manner or to determine the position of objects from complete images in a simple manner, wherein the performance of a correspondingly trained conditional neural process can be optimized even further.
With a further example embodiment of the present invention, a control device for determining a position of an object is moreover also specified, wherein the control device comprises a provisioning unit designed to provide image data, wherein the image data comprise target image data showing the object and labeled comparison image data regarding the object; a reception unit designed to receive a conditional neural process, trained by a control device described above for training a conditional neural network for determining an object from image data, for determining a position of an object from image data; and a determination unit designed to determine, by means of the provided conditional neural process for determining a position of an object from image data, the position of the object based on the provided image data.
Such a control device for determining a position of an object has the advantage that it is based on a conditional neural process, trained by an improved control device for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data, for determining a position of an object from image data. In particular, the combination of functional contrastive learning and an end-to-end learning approach in the training of the conditional neural process has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks. Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
With a further example embodiment of the present invention, a control device for controlling a controllable system is furthermore also specified, wherein the control device comprises a reception unit designed to receive a position of an object determined by a control device described above for determining a position of an object; and a control unit designed to control the controllable system based on the determined position of the object.
Such a control device for controlling a controllable system has the advantage that it is based on a conditional neural process, trained by an improved control device for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data, for determining a position of an object from image data. In particular, the combination of functional contrastive learning and an end-to-end learning approach in the training of the conditional neural process has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks. Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
In summary, it can be noted that the present invention provides a method for training a conditional neural process for determining a position of an object from image data, with which a conditional neural process for determining a position of an object from image data with optimized performance can be trained with comparatively low resource consumption.
The described embodiments and developments of the present invention can be combined with one another as desired.
Further possible embodiments, developments and implementations of the present invention also include not explicitly mentioned combinations of features of the present invention described above or below with respect to exemplary embodiments.
The figures are intended to provide a further understanding of example embodiments of the present invention. They illustrate example embodiments and, in connection with the description, serve to explain principles and concepts of the present invention.
Other embodiments and many of the mentioned advantages become apparent from the figures. The illustrated elements of the figures are not necessarily shown to scale with respect to one another.
In the figures, identical reference signs denote identical or functionally identical elements, parts or components, unless stated otherwise.
The present invention relates to a method for training a conditional neural process for determining a position of an object from image data, and in particular to a method for training a conditional neural process for determining a position of an object from image data, with which a conditional neural process for determining a position of an object from image data with optimized performance can be trained with comparatively low resource consumption.
The term “meta-learning algorithm” is understood to mean a machine learning algorithm designed to optimize the algorithm by autonomous learning as well as drawing on experiences. Such meta-learning algorithms are in particular applied to metadata, wherein the metadata may, for example, be properties of the corresponding learning problem, algorithm properties, or patterns previously derived from the data. The application of such meta-learning algorithms in particular has the advantage that the performance of the algorithm can be increased and the algorithm can be adapted quickly and flexibly to different problems and/or new categories of objects. Such meta-learning algorithms are used, for example, to determine a position and/or pose, or 6D-pose, of an object based on image data.
Meta-learning algorithms include, for example, model-agnostic meta-learning (MAML) or conditional neural processes. The aim of these algorithms is to optimize model parameters in such a way that training success can be achieved with comparatively few gradient optimizations. Conditional neural processes are in particular based on using a feed-forward neural network to calculate the training data information, to aggregate this information, and to route this information to another feed-forward network for inference.
However, it proves disadvantageous with such meta-learning algorithms, for example, that the training of such algorithms is comparatively complex and can lead to so-called overfitting or memorization of training data. In particular, during the training of such an algorithm, a state can occur in which only problem solutions determined from the training data are reproduced, that is, the algorithm correctly processes only the training data and does not achieve any new results when new data are input.
The combination of functional contrastive learning and an end-to-end learning approach in particular has the advantage that the performance of the correspondingly trained conditional neural process, and in particular the accuracy in determining the position of an object, can be optimized, which proves advantageous in particular for specific practical tasks.
Moreover, the conditional neural process can be trained with comparatively low resource consumption, in particular with comparatively low memory and processor capacities, especially since the individual representations are coordinated with one another.
Specified overall is thus an improved method for training a meta-learning algorithm and in particular a conditional neural process for determining a position of an object from image data 1.
In this respect, it has also been shown that better performance can in particular be achieved with a thus trained conditional neural process than with comparable model-agnostic meta-learning.
The amount of image data showing a particular object may also be different from the amount of corresponding comparison data, wherein these amounts may also differ depending on the application or task.
The method may furthermore also comprise a step of capturing current image data showing the particular object, wherein the captured image data can be processed correspondingly and can subsequently be provided as image data showing the particular object.
According to the embodiments of
As
According to the exemplary embodiments of
The image data and the comparison image data respectively are image data showing complete images, wherein the image data may in particular be higher-dimensional image data.
The trained conditional neural process may subsequently be utilized, for example, to determine a position and/or a pose of an object in image data. Furthermore, the trained conditional neural process may however also be used to recognize abnormalities in image data, for example.
The determined position and/or pose of the object may subsequently be used, for example, to control a controllable system, for example, to control a robot arm to grip the object. Furthermore, the determined position or pose may however also be used, for example, to control or navigate an autonomous vehicle based on an identified target vehicle or for facial recognition.
As
According to the embodiments of
The provisioning unit may, for example, be a receiver designed to receive the image data, for example from one or more optical sensors. The training unit may furthermore be implemented, for example, based on code that is stored in a memory and can be executed by a processor.
As
The first generation unit, the second generation unit and the first determination unit can in turn be respectively implemented, for example, based on code that is stored in a memory and can be executed by a processor.
As
The second determination unit, the third determination unit and the fourth determination unit can in turn be respectively implemented, for example, based on code that is stored in a memory and can be executed by a processor.
Furthermore, the image data and the comparison image data in turn respectively are image data showing complete images.
According to embodiments of
The further provisioning unit and the further reception unit may each, for example, be appropriately designed receivers. Furthermore, the further determination unit may in turn be implemented, for example, based on code that is stored in a memory and can be executed by a processor.
According to the embodiments of
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2022 202 030.8 | Feb 2022 | DE | national |
