Patent Application
20230201973
The present invention relates to a system and a method for automating welding processes, in particular welding processes in the heavy industries.
In the manufacturing industry, robots are used to perform accurate, highly precise operations every day. Many of these industrial robots are programmed to perform the same exact motions and to repeat these many times a day. Accordingly, robotic welding systems are commonly used to accurately and repeatedly weld components together in industries like the automotive industry as well as in heavy industry, such as in shipyards. Whereas welding applications in the automotive industries is dominated by pre-programmed welding programs, the welding processes in the heavy industries are dominated by tasks that differ between each run as the welding operations are complex with huge tolerances of the components. This means that sensing and detection of the welding task online is an essential requirement of the automation of the process.
One of the challenges in robotic welding applications is to find out where the welding process is to happen and what the welding path for a particular welding task is. The optimal welding path depends on the type of objects and features present in a welding task, objects such as profiles, bars, stiffeners, brackets, collar plates, inserts, cutouts, waterholes, welding seams, plate connections, chamfers, tacks, gaps, plate thickness, bevel, scallop, etc.
CN110227876 and CN 110524581 discloses methods for autonomously planning a robot welding path based scanning of a known welding object to obtain on 3D point cloud data and compare to a 3D point cloud data of a CAD model of the known welding object in order to obtain the complete work piece weld information, extract the weld pose information and process it to plan the robot welding path.
Often the welding objects of present and subsequent welding tasks are not known. The purpose of the present disclosure is therefore to be able to plan the robot welding path for robotic welding in a complex welding environment, such as in the heavy industries.
The present disclosure therefore relates to a method for automatic detection of a welding task in a welding environment. Once the welding task has been determined it is possible to plan and generate a welding path of the welding objects in the welding environment, which can be executed by a welding robot. Generally speaking the presently disclosed system and method relates to automatic detection of a welding task in a welding environment by obtaining scanning data of the welding environment and using artificial intelligence (AI), such as deep learning and/or neural networks, to detect and identify/recognize welding objects in the welding environment. This relies on the fact that even though the specific objects to be welded in present and subsequent welding tasks are not known, the group of possible welding objects are known. I.e. once scanning data of the welding environment has been obtained, it will be a matter of recognizing the welding objects in the welding environment. But that is not a simple task and utilization of AI is one way to achieve that.
One embodiment of the present disclosure therefore relates to a computer implemented method for automatic detection and/or planning of a welding task in a welding environment, the method comprising the steps of:
obtaining scanning data from a scan of the welding environment,
detecting welding object(s) in the scanning data, and identifying the type of the welding object(s) by means of artificial intelligence, such as machine learning, wherein the machine learning algorithm has been trained on real and simulated 3D data of known welding objects,
determining the pose of each detected welding object, and
optionally generating a welding path for each detected welding object.
Another embodiment of the present disclosure relates to a computer implemented method for automatic detection and/or planning of a welding task in a welding environment, the method comprising the steps of:
obtaining scanning data from a 3D scan of the welding environment,
detecting welding object(s) in the scanning data by means of artificial intelligence employing a supervised learning algorithm, wherein the supervised learning algorithm has been trained on real and simulated 3D data of known welding objects,
determining the pose of each detected welding object, and
optionally generating a welding path for each detected welding object.
Once welding objects are detected in the scanning data by means of AI it is possible to plan an associated welding task because the welding objects are now known. Planning of a welding task typically includes generating a welding path for each detected welding object. It is also preferred to determine the pose of each detected welding object once a welding object is detected. The presently disclosed method can thereby automatically detect and/or identify the types of objects within the weld environment in order to determine which components and along which path the welding process should be executed.
A key aspect of the present invention is that the AI algorithm, e.g. a machine learning algorithm, supervised or unsupervised, has been trained on 3D data of known welding objects, real 3D data and/or simulated 3D data, preferably real and simulated 3D data of known welding objects and real and simulated data of the background, e.g. the welding environment. Generally, it would be very time consuming to train a machine learning algorithm using only real 3D data e.g. obtained from a 3D scan of known welding objects. The inventors have realised that by combining real 3D data with simulated 3D data (e.g. point clouds generated from CAD models of known welding objects), much more training material can be provided in a shorter period. It is even possible to detect welding objects in the welding environment based merely on 2D data, e.g. 2D images obtained from 2D scanning of the welding environment, which has been provided as input to the machine learning algorithm. However, it is preferred to use 3D data as input to the machine learning algorithm.
The invention will in the following be described in greater detail with reference to the accompanying drawings:
The term “welding environment” refers herein to the area of which the welding task is to happen, hence the area in which all objects of a welding task are placed. A welding task may comprise one or more welding objects and hence one or more welding objects may be placed in the welding environment.
Welding objects refer to the objects of a given welding task. Thus, not all welding objects should necessarily be welded, as some welding objects/objects of a welding task may be drain holes or other objects to avoid in the welding process. Their detection and pose is still important when determining the welding path as they may also indicate areas to avoid in the welding path.
Artificial intelligence can be employed to identify a specific context or action, or generate a probability distribution of specific states of a system without human intervention. Artificial intelligence relies on applying advanced mathematical algorithms, e.g. decision trees, neural networks, regression analysis, cluster analysis, genetic algorithms, and reinforced learning, to a set of available data (information) on the system. The artificial intelligence techniques can be used to perform determinations disclosed herein.
Artificial neural networks (ANN) or just “neural network” as used herein, or connectionist systems are computing systems inspired by the biological neural networks that constitute animal brains. Such systems learn (progressively improve their ability) to do tasks by considering examples, generally without task-specific programming. For example, in image recognition, they might learn to identify images that contain cats by analyzing example images that have been manually labeled as “cat” or “no cat” and using the analytic results to identify cats in other images. They have found most use in applications difficult to express with a traditional computer algorithm using rule-based programming. An ANN is based on a collection of connected units called artificial neurons, i.e. modelling the neurons in a brain. Like the synapses in a biological brain each connection can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it.
Deep learning (also known as deep structured learning or differential programming) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised. Deep learning architectures such as deep neural networks, deep belief networks, recurrent neural networks and convolutional neural networks have been applied to fields including computer vision, as also used herein.
A deep neural network is an artificial neural network with multiple layers between the input and output layers. A deep neural network finds the correct mathematical manipulation to relate the input into the output, whether it be a linear relationship or a non-linear relationship. The network moves through the layers calculating the probability of each output.
The presently disclosed approach may be based on a neural network or a deep neural network. Both terms are used interchangeably in the present disclosure. As used herein, the term “neural network” refers to an interconnected group of natural or artificial neurons that uses a computational/mathematical model for information processing based on a connectionistic approach to computation. Neural networks are adaptive systems that change structure based on external or internal information that flows through the network. They are used to implement non-linear statistical data modelling and may be used to model complex relationships between inputs and outputs. In this case the input can be point cloud data and the output can be detection of welding objects in the input point cloud data.
As used herein, the term “point cloud” generally refers to a three-dimensional set of points forming a three-dimensional view of a subject reconstructed from a number of two-dimensional views. In a three-dimensional image capture system, a number of such point clouds may also be registered and combined into an aggregate point cloud constructed from images captured by a moving camera, e.g. a scanner. In this approach point clouds can be generated from CAD data, e.g. CAD data of known welding objects and thereby be used to generate and represent simulated data. But point clouds can also be generated from real 3D scan data, of e.g. welding objects, background data, welding environments, etc., and thereby represent real data.
In computer vision, image segmentation is the process of partitioning a digital image into multiple segments in order to simplify and/or change the representation of an image into something that is more meaningful and easier to analyse, aka image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics. Semantic segmentation then refers to the process of linking each pixel in an image to a class label, i.e. semantic image segmentation is the task of classifying each pixel in an image from a predefined set of classes.
Computing the pose (aka 3D pose) of a rigid body with respect to a camera is a well-studied problem in computer/robot vision. A 3D pose can be solved by starting with the known features of an object and matching these features with their 2D correspondence in the image. Features such as point and line segments are commonly used. In computer vision “pose estimation” may be the determination of the 3D pose of a rigid body from a single 2D image. If using point-based correspondences, the problem is known as “perspective-n-point,” where n is the number of correspondences. Three non-collinear points provides four solutions. Four or more non-collinear points provides a unique solution. Alternatively pose estimation may be provided by means of data acquired from a time-of-flight camera.
In one embodiment of the present disclosure at least one scanner is used to acquire and generate scanning data of the welding environment in order to detect the objects to be welded. The scanner may be a 1D scanner, e.g. a line scanner, a 2D scanner, e.g. a standard camera, or a 3D scanner. The scanner may be part of the presently disclosed welding system. The scan data may be processed prior to it being used as input data to the AI algorithm, e.g. deep learning algorithm, used to detect and recognise shapes and/or objects and/or parts of objects which should be welded. The data from the scanner can be in form of frame data, which can be used to form 2D data to input to the supervised learning algorithm. But frame data can also be used to generate a 3D point cloud which can be used as input to the AI algorithm, e.g. a supervised learning algorithm. However, 3D point cloud data can also be directly outputted from the scanner. But the scan data will typically be unordered, e.g. an unordered 3D point cloud.
The object detection may be provided by using an AI/machine learning based tool, such as an AI/machine learning based algorithm, e.g. neural network or a deep neural network, to detect the welding objects, e.g. identify and determine the type of objects within the welding environment. An AI based approach typically relies on a trained model, and a trained model in the presently disclosed approach can preferably be trained on both real and simulated data to enable identification of welding objects such as profiles, bars, stiffeners, brackets, collar plates, inserts, cutouts, waterholes, welding seams, plate connections, chamfers, tacks, gaps, plate thickness, bevel, scallop, etc. Hence the AI based approach, e.g. neural network and/or deep learning, may learn the shapes of these objects through the training phase creating a representation of the distinguished features of the welding object, e.g. within the neural network. This representation may then be used in the detection phase to detect and identify the object, the location of the object and/or the pose of the object in the scan data. This information can then be used to plan the welding task.
As also stated above the learning/training of the model used can be supervised, semi-supervised or unsupervised. The most practical implementation is to use supervised learning because it is less time and resource consuming and it requires less data to supervise the model during training. In supervised learning subsidiary goals/teaching are introduced in several parts of the welding process, also during the detection and identification phase. However, in this case it has been realized that the combination of real data and simulated data can greatly increase the amount of relevant training data and therefore it has been shown to be possible to use unsupervised learning in the presently disclosed approach, where the model for example is trained by merely feeding an end goal into the training phase. E.g. an end goal can merely be a “good weld” and during the training a plurality of random variations is fed into the model thereby examining when a “good weld” is obtained. During such unsupervised learning the machine learning model's ability to detect and identify welding objects will also improve because a good weld cannot be obtained if the welding object is not detected and identified correctly. And in that regard the presently disclosed approach of combining real and simulated data in the training of the AI model has turned out to be a key factor.
The ratio between real data and simulated data has an influence on the performance of the presently disclosed automatic welding process. A success ratio of around 90% in automatic detection and identification of the welding objects can be acceptable in some setups and such a success ratio of around 90% can be obtained with a ratio of approx. 10% real data and 90% simulated data. Success ratios of more than 90%, e.g. 95% or close to 100% success ratio, requires more real data as input to the training phase. However, is has been shown that with a ratio of approx. 30% real data and 70% simulated data close to 100% success ratio can be obtained in welding object detection and identification. I.e. the robustness of the presently disclosed approach may scale with the ratio of real data and simulated data in the training.
The amount of data required as input to the training phase scales with the number of different welding objects that must be discriminated and identified. If only a few welding objects, e.g. less than 5 or 10 different welding objects, can be identified the amount of necessary real training data is limited to only a few real scans. However, if more than 10, or 50 or 100 or even thousands of welding objects must be discriminated during the automatic welding process, the amount of data going into the training also increases. I.e. the robustness of the presently disclosed approach may also scale with the number of welding objects that can be identified.
In a real setup the requirement regarding robustness of the identification and the number of possible welding objects are typically known, and then it will be a matter of collecting the suitable amount of real data and generating a suitable amount of simulated data for the training, to meet the robustness requirement.
Thus, it is possible to identify location and pose of each object of a welding task by applying an AI algorithm trained to recognise the individual welding objects in the point cloud data. This information may then be used to plan the welding task.
In the preferred embodiment of the present disclosure the object detection is used to identify the type/category of welding object such as profiles, bars, stiffeners, brackets, collar plates, inserts, cutouts, waterholes, welding seams, plate connections, chamfers, tacks, gaps, plate thickness, bevel, scallop, etc. Each of these exemplary categories of welding objects can have different types. Depending on the type of objects detected in a given welding task the method may determine the most suitable welding path such that some areas, such as waterholes, are avoided and the proper section of for example a bracket is welded.
Frame data from a scan of the welding environment can be used to generate a stitched scene-data from the individual frames. Advantageously the data points of the stitched scene-data from the individual frames are down-sampled or compressed prior to the generation of an organised 3D point cloud. In the preferred embodiment the outlying data points of the stitched scene-data from the individual frames are removed prior to the generation of the organised point cloud, such as by the use of random sample consensus. This may reduce or even prevents the use of computer power used to identify the nature of false-read outlier points.
In one embodiment the points of the 3D point cloud are segmented prior to object detection.
In the preferred embodiment of the present disclosure the AI approach is configured to recognise shapes and/or objects and/or part of objects to be welded in a given welding task. By object recognition it is possibly to identify the objects which should be welded and from the pose and position of the detected objects it is possible to determine the proper welding path. Some objects are to be avoided all together and other object has designated areas in which the welding should happen, and hence determination of the welding path may be straightforward once welding objects are detected by the presently disclosed method.
The learning/training phase may be provided exclusively on simulated data or exclusively on real data, i.e. scan data acquired from real physical objects. However, the best detection is provided with if the learning/training phase is based on combination of real data and simulated data, because the amount of training data can be greatly increased when using simulated data because such kind of simulated data can be automatically multiplied, whereas real data is usually necessary to improve the robustness of the object detection. The amount of real data is preferably between 5-50%, more preferably 10-40%, most preferably 10-30%, where the rest is simulated data. Simulated data can be point clouds generated from CAD models of the known welding objects, or virtual backgrounds or virtual welding environments.
When training the model, e.g. a neural network, both real data and simulated data can advantageously be used. Real data from real physical welding environments may initially be difficult to obtain, whereas simulated training data can more easily be generated. By combining real data with simulated data in the training phase, a sufficient amount of training data can be provided for the training phase.
Another example of the presently disclosed approach is illustrated in
The compressed point cloud dataset may then be fed into an unsupervised or supervised AI algorithm, e.g. a deep neural network, for example by using the PointNet architecture, which is an example of applying machine learning directly on point clouds.
One advantage of the PointNet approach is that instead of transforming the point cloud data into regular 3D voxel grids or collections of images. This makes it highly efficient and effective and thereby applicable in a real and complex environment as the present welding environment. The present disclosed approach can therefore take point clouds as input to a supervised AI algorithm and the output is labelled for each point of the input, which again can be processed as part of the supervised AI algorithm, such that point labels become semantic object classes such that detection of 3D objects, e.g. welding objects, can be directly provided therefrom. Hence, the AI algorithm can be trained to recognize different objects such as profiles, bars, stiffeners, brackets, collar plates, inserts, cutouts, waterholes, welding seams, plate connections, chamfers, tacks, gaps, plate thickness, bevels or scallops within the welding environment.
The output from the supervised or unsupervised learning algorithm is passed to a post-processing step comprising pose estimation and/or location of each detected welding object in the welding environment object. This makes it possible generate a welding path for each object which can be executed by a welding system.
The present disclosure further relates to a welding system comprising a welding machine used to weld material together in an automatic or semi-automatic manner. A robotic or similar automated motion generating mechanism (hereafter referred to as a robot) moves the welding gun of the welding machine while welding the material. The welding machine and the robot are controlled by a robot controller. A scanner may also be part of the welding system, e.g. a 2D scanner or a 3D scanner. A welding path generated by the presently disclosed method can preferably be executed by the presently disclosed welding system. However, the presently disclosed method may also be an integral part of the presently disclosed welding system such that the welding system autonomously can detect, plan and execute welding tasks in a complex environment.
The present disclosure further relates to a system for automatic detection and/or planning of a welding task in a welding environment, comprising a non-transitive, computer-readable storage device for storing instructions that, when executed by a processor, performs a method for automatic detection and/or planning of a welding task in a welding environment according to the described method. The system may comprise a computing device comprising a processor and a memory and being adapted to perform the method, but it can also by a stationary system or a system operating from a centralized location, and/or a remote system, involving e.g. cloud computing. In this case the actual processing, in particular the detection of welding objects based on AI, may be provided by cloud computing.
The present disclosure further relates to a computer program having instructions which when executed by a computing device or system cause the computing device or system to automatically detect and/or plan a welding task in a welding environment according to the described method. Computer program in this context shall be construed broadly and include e.g. programs to be run on a PC or software designed to run on welding systems, smartphones, tablet computers or other mobile devices. Computer programs and mobile applications include software that is free and software that has to be bought, and also include software that is distributed over distribution software platforms.
| Number | Date | Country | Kind |
|---|---|---|---|
| 20176332.3 | May 2020 | EP | regional |
This application is the U.S. National Stage of PCT/EP2021/063889 filed on May 25, 2021, which claims priority to European Patent Application 20176332.3 filed on May 25, 2020, the entire content of both are incorporated herein by reference in their entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/063889 | 5/25/2021 | WO |
