Patent Application
20240424675
The present disclosure relates to a working state determination device, a working state determination method, and a working state determination program.
In assembly work by a robot at a manufacturing site, a technique of simplifying adjustment work without using a jig or the like by advancing intelligence of the robot has been studied. For example, a determination method using a Recurrent Neural Network (RNN) for assembly work of a work having a snap-fit has been proposed (see Non Patent Literature 1). In this method, attention is paid to a transient change of a robot arm tip reaction force when a workpiece is fitted, and the RNN is configured and learned with a small time duration in which the transient waveform is captured.
Furthermore, for example, a robot system that determines whether work is good or bad depending on various causes in fitting and insertion work has been proposed (see Patent Literature 1). In this system, a feature amount existence range at the time of successful work and a feature amount existence range at the time of failure are calculated for each failure cause, an existence area of each feature amount is recorded, and work quality and a cause at the time of failure are determined depending on which feature amount existence area the feature amount at the time of work execution is in.
In the related art, it is possible to determine whether assembly work is completed or not completed. However, in an actual manufacturing site, for example, there is also an abnormal system in which assembly work cannot be continued unless regrasping work of a workpiece is performed. In the prior art, there is a problem that such an abnormal system cannot be determined or a determiner needs to be adjusted on a one-by-one basis by feature engineering.
The present disclosure has been made in view of the above points, and an object thereof is to determine a working state including a normal system and an abnormal system.
In order to achieve the above object, a working state determination device according to the present disclosure includes an acquisition unit configured to acquire data detected by a force sensor that detects a force acting on a hand tip portion of a robot hand, and a determination unit configured to determine a working state of the robot hand by inputting data acquired by the acquisition unit to a determiner that determines the working state of the robot hand and in which a first determiner that extracts a feature of a normal working state and a second determiner that extracts a feature of an abnormal working state are coupled, the determiner outputting three states as the working state, the three states being a work completion state, a work continuation state, and a work continuation impossible state.
In addition, time-series data acquired by the acquisition unit may be input to the determiner, and a time duration of data input to the second determiner may be longer than a time duration of data input to the first determiner.
The time duration for the first determiner may be a time duration corresponding to a transient response of data generated at the time of work completion, and the time duration for the second determiner may be a time duration corresponding to a time until an abnormality occurs and the operation of the robot hand is stopped.
In addition, the first determiner may be learned by using learning data in which each piece of data before transition of the time-series data in a case in which the working state transitions from work continuation to work completion is labeled as work continuation and the data after transition is labeled as work completion, and the second determiner may be learned by using learning data in which data before transition of the time-series data in a case in which the working state transitions from work continuation to work continuation impossible is labeled as work continuation and the data after transition is labeled as work continuation impossible.
The working state determination device according to the present disclosure may further include a learning unit that executes learning of the determiner using the learning data.
Furthermore, the learning unit may generate pseudo learning data using the learning data and a random number, add the pseudo learning data to the learning data, and use the pseudo learning data for learning by the determiner.
The acquisition unit may further acquire a position of the hand tip portion of the robot hand, and the determination unit may determine the working state by inputting data detected by the force sensor and the position of the hand tip portion to the determiner.
Furthermore, the determiner may include a first neural network constituting the first determiner and a second neural network constituting the second determiner, and may output a result obtained by combining an output of the first neural network and an output of the second neural network.
The working state determination device according to the present disclosure may further include a control unit that controls the operation of the robot hand on the basis of the working state determined by the determination unit.
The working state determination method according to the present disclosure is a method in which an acquisition unit acquires data detected by a force sensor that detects a force acting on an hand tip portion of a robot hand, and a determination unit determines a working state of the robot hand by inputting data acquired by the acquisition unit to a determiner that determines the working state of the robot hand and in which a first determiner that extracts a feature of a normal working state and a second determiner that extracts a feature of an abnormal working state are coupled, the determiner outputting three states as the working state, that is, a work completion state, a work continuation state, and a work continuation impossible state.
The working state determination program according to the present disclosure is a program for causing a computer to function as an acquisition unit configured to acquire data detected by a force sensor that detects a force acting on an hand tip portion of a robot hand, and as a determination unit configured to determine a working state of the robot hand by inputting data acquired by the acquisition unit to a determiner that determines the working state of the robot hand and in which a first determiner that extracts a feature of a normal working state and a second determiner that extracts a feature of an abnormal working state are coupled, the determiner outputting three states as the working state, the three states being, a work completion state, a work continuation state, and a work continuation impossible state.
According to the working state determination device, the method, and the program according to the present disclosure, the working state including the normal system and the abnormal system can be determined.
Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent components and portions are denoted by the same reference numerals. In addition, dimensions and ratios of the drawings are exaggerated for convenience of description, and may be different from actual ratios.
As illustrated in
The manipulator 52 is, for example, a vertical articulated robot having a configuration of six degrees of freedom necessary for operation in a three-dimensional space. The degree of freedom of the manipulator 52 may be seven degrees of freedom obtained by adding a redundant degree of freedom to six degrees of freedom. The manipulator 52 drives each joint such that a rotation angle of each joint becomes an angle corresponding to control information (details will be described later) output from the working state determination device 10. As a result, the gripper 56 is controlled to be at the designated position and orientation. In addition, the manipulator 52 outputs the current rotation angle of each joint to the working state determination device 10.
The gripper 56 preferably has versatility capable of gripping various workpieces in various gripping states, and has a configuration for preventing large gripping deviation. For example, the gripper 56 may be a parallel two-finger gripper in which non-slip rubber is attached to a surface for gripping the workpiece.
The force sensor 54 is installed between the wrist portion of the manipulator 52 and the gripper 56, detects external force values in six directions of three parallel axes and three rotation axes acting on the gripper 56, and outputs the values as force sensor values. The force sensor value is an example of “data detected by the force sensor” of the disclosed technology.
The working state determination device 10 controls the operation of the manipulator 52 while determining the working state by the manipulator 52. In the present embodiment, as illustrated in
In the present embodiment, as the working state to be determined, work completion, work continuation impossible, and work continuation are set as illustrated in
The storage device 16 stores a working state determination program for executing learning processing and determination control processing to be described later. The CPU 12 is a central processing unit, and executes various programs and controls each configuration. That is, the CPU 12 reads the program from the storage device 16 and executes the program using the memory 14 as a work area. The CPU 12 performs control of each of the above-described configurations and various types of arithmetic processing according to a program stored in the storage device 16.
The memory 14 includes a random access memory (RAM), and temporarily stores a program and data as a work area. The storage device 16 includes a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), and the like, and stores various programs including an operating system and various data.
The input device 18 is a device for performing various inputs, such as a keyboard and a mouse. The output device 20 is, for example, a device for outputting various types of information, such as a display and a printer. A touch panel display may be employed as the output device 20 to function as the input device 18. The storage medium reading device 22 reads data stored in various storage media such as a compact disc (CD)-ROM, a digital versatile disc (DVD)-ROM, a Blu-ray disc, and a universal serial bus (USB) memory, writes data to the storage medium, and the like. The communication I/F 24 is an interface for communicating with other devices, and for example, standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used.
Next, a functional configuration of the working state determination device 10 according to the first embodiment will be described.
The acquisition unit 32 acquires learning data input to the working state determination device 10 at the time of learning processing. The learning data is data in which the force sensor value detected by the force sensor 54 at each time and the information on the arm tip position of the gripper 56 at each time are associated with the label indicating the correct answer of the working state at each time. The information on the arm tip position of the gripper 56 (hereinafter, also simply referred to as “arm tip position”) may be an absolute position of a predetermined position such as the tip of the gripper 56, or may be a relative movement amount or an absolute movement amount of a predetermined position during one time step. The arm tip position is obtained by matrix calculation based on the rotation angle (hereinafter, referred to as “joint angle value”) of each joint output from the manipulator 52 and the kinematics information of the manipulator 52.
Here, generation of learning data will be described. As described above, in the present embodiment, three working states of work completion, work continuation impossible, and work continuation are determined. Therefore, when the working state transitions from the work continuation to the work completion, as indicated by a broken line in the left diagram of
Furthermore, when the working state transitions from the work continuation to the work continuation impossible, as indicated by a broken line in the right diagram of
When the learning data is generated as described above, the number of pieces of learning data to which each label of work completion and work continuation impossible is given is smaller than the number of pieces of learning data to which the label of work continuation is given, and the number of pieces of learning data is biased. Therefore, learning data of a small number of labels may be copied and increased so that the number of pieces of learning data for each label is about the same. As a result, over-learning can be prevented, and generalization performance can be prevented from being impaired.
When acquiring the learning data, the acquisition unit 32 passes the acquired learning data to the learning unit 34. Hereinafter, learning data in a case where the working state transitions from work continuation to work completion is referred to as “normal system learning data”, and learning data in a case in which the working state transitions from work continuation to work continuation impossible is referred to as “abnormal system learning data”.
In addition, the acquisition unit 32 acquires the force sensor value output from the force sensor 54 and the joint angle value output from the manipulator 52 during the determination control processing. The acquisition unit 32 calculates the arm tip position by matrix calculation on the basis of the acquired joint angle value and the kinematics information of the manipulator 52. The acquisition unit 32 associates the acquired force sensor value with the calculated arm tip position to obtain target data, and delivers the target data to the determination unit 38.
The learning unit 34 uses the learning data transferred from the acquisition unit 32 to execute learning of parameters of the determiner 36 that determines the working state of the manipulator 52. The determiner 36 may include, for example, a deep neural network (DNN). In the present embodiment, a case in which the determiner 36 is configured by a Recurrent Neural Network (RNN) for time-series data will be described.
A weighted value of the value output from each neuron of the corresponding input layer is input to each neuron of the intermediate layer. In addition, a value weighted to a value output from a neuron in the intermediate layer of one time ago is input to each neuron in the intermediate layer. Each neuron in the intermediate layer outputs a value obtained by applying the activation function to the input value to the neuron in the intermediate layer of one time later. For example, each neuron in the intermediate layer may output the following vector value hi(t).
Ui is a parameter representing a weight applied to a value output from each neuron of the input layer, and Wi is a parameter representing a weight applied to a value output from each neuron of the intermediate layer of one time ago. Furthermore, tanh is an example of an activation function, and bi is a parameter representing bias. The output h1(T_1) of the intermediate layer of the first determiner 36 is a vector value obtained by characterizing the force sensor value and the arm tip position at time t=1 to t=T1. The output h2(T_2) of the intermediate layer of the second determiner 36B is a vector value obtained by characterizing the force sensor value and the arm tip position at times t=1 to t=T2.
For example, as described below, the output layer combines the output from the first determiner 36A and the output from the second determiner 36B, and outputs a determination result.
Vi is a weight related to the output hi(T_i) from the neuron of the intermediate layer of t=Ti, c is a parameter, and softmax is an example of an activation function. y{circumflex over ( )}(in
Here, the time duration Ti(i∈[1;2) to be input to each of the first determiner 36A and the second determiner 36B is described. As described above, in a case in which the working state transitions from the work continuation to the work completion, as illustrated in the left diagram of
The learning unit 34 determines the parameters of the determiner 36 by executing the learning of the determiner 36 as described above using the learning data delivered from the acquisition unit 32. Specifically, the learning unit 34 first determines T1 and T2. For example, the learning unit 34 may determine T1 and T2 by receiving T1 and T2 designated by the user on the basis of the learning data. Furthermore, the learning unit 34 may determine T1 and T2 that can obtain the most accurate output by executing learning while changing the time duration of each of T1 and T2 by a predetermined value.
The learning unit 34 sets the determined T1 and T2 in the first determiner 36A and the second determiner 36B, respectively, and adds a provisional output layer as described below to the output hi(T-i) of the intermediate layer.
The learning unit 34 inputs the normal system learning data to the first determiner 36A to which the temporary output layer is added, and inputs the abnormal system learning data to the second determiner 36B to which the temporary output layer is added. Then, the learning unit 34 determines the parameters Ui, Wi, and bi included in the input layer and the intermediate layer of the determiner 36 so as to minimize the difference (for example, cross entropy or the like) between the output value yi{circumflex over ( )} and the label given to the learning data.
Furthermore, the learning unit 34 inputs the learning data to the determiner 36 after fixing Ti, Ui, Wi, and bi determined as described above. Then, the learning unit 34 determines the parameters Vi and c included in the output layer so as to minimize the difference (for example, cross entropy or the like) between the output value y{circumflex over ( )} and the label given to the learning data. The learning data input to the determiner 36 here may be learning data different from the learning data used when the parameters of the input layer and the intermediate layer are determined. In addition, the normal system learning data and the abnormal system learning data used when the parameters of the input layer and the intermediate layer are determined may be randomly mixed and used.
The learning unit 34 stores the determiner 36 in which the parameters Ti, Ui, Wi, bi, Vi, and c are determined as described above in a predetermined storage area.
The determination unit 38 inputs the target data delivered from the acquisition unit 32 to the determiner 36, determines the current working state, and delivers the determination result to the control unit 40.
The control unit 40 determines the action of the manipulator 52 on the basis of the working state indicated by the determination result delivered from the determination unit 38. For example, the control unit 40 may determine an action on the basis of a predetermined action rule of the manipulator 52 according to the working state. As the action rule, for example, in a case in which the working state is work completion, returning to the initial posture, in a case in which the working state is work continuation impossible, regrasping of the main work 60, and the like may be determined. The control unit 40 generates control information for causing the manipulator 52 to execute an action corresponding to the determined action. Specifically, the control information is a rotation angle of each joint for achieving the posture of the manipulator 52 in the determined action. The control unit 40 outputs the generated control information to the manipulator 52.
Next, the operation of the robot system 100 according to the present embodiment will be described.
When learning data is input to the working state determination device 10 and learning of the determiner 36 is instructed, the working state determination device 10 executes learning processing. Then, when work by the manipulator 52 is started in a state where the determiner 36 on which learning has been executed is stored in a predetermined storage area, determination control processing is executed in the working state determination device 10. The learning processing and the determination control processing are examples of a “working state determination method” of the disclosed technology.
In step S10, the acquisition unit 32 acquires the learning data including the normal system and the abnormal system input to the working state determination device 10 and delivers the learning data to the learning unit 34. Next, in step S12, the learning unit 34 determines the time duration T1 of the first determiner 36A and the time duration T2 of the second determiner 36B by receiving T1 and T2 from the user.
Next, in step S14, the learning unit 34 sets the determined T1 and T2 in the first determiner 36A and the second determiner 36B, respectively, and adds a temporary output layer to the output hi(T-1) of the intermediate layer. The learning unit 34 inputs the normal system learning data to the first determiner 36A to which the temporary output layer is added, and inputs the abnormal system learning data to the second determiner 36B to which the temporary output layer is added. Then, the learning unit 34 determines the parameters Ui, Wi, and bi (i∈[1;2) included in the input layer and the intermediate layer of the determiner 36 so as to minimize the difference between the output value of the temporary output layer and the label given to the learning data.
Next, in step S16, the learning unit 34 inputs the learning data to the determiner 36 after fixing Ti, Ui, Wi, and bi determined in steps S12 and S14. Then, the learning unit 34 determines the parameters Vi and c included in the output layer so as to minimize the difference between the output value of the output layer and the label given to the learning data.
Next, in step S18, the learning unit 34 stores the determiner 36 in which the parameters Ti, Ui, Wi, bi, Vi, and c have been determined in steps S12 to S16 in a predetermined storage area, and the learning processing ends.
In step S20, the acquisition unit 32 acquires the force sensor value output from the force sensor 54 and the joint angle value output from the manipulator 52. Next, in step S22, the acquisition unit 32 calculates the arm tip position by matrix calculation based on the acquired joint angle value and the kinematics information of the manipulator 52.
Next, in step S24, the acquisition unit 32 associates the force sensor value acquired in step S20 with the arm tip position calculated in step S22 to obtain target data, and delivers the target data to the determination unit 38. The determination unit 38 shifts the vector value xi(t) input to each neuron of the input layer of the determiner 36 to a neuron of one time ago, and inputs the target data delivered from the acquisition unit 32 to a neuron of the input layer corresponding to xi(t). Then, the determination unit 38 determines the current operation state on the basis of the output value from the output layer of the determiner 36, and passes the determination result to the control unit 40.
Next, in step S26, the control unit 40 determines the action of the manipulator 52 on the basis of the working state indicated by the determination result delivered from the determination unit 38. Next, in step S28, the control unit 40 generates control information for causing the manipulator 52 to execute an action corresponding to the determined action, and outputs the control information to the manipulator 52. Then, the determination control processing ends.
As described above, in the robot system according to the present embodiment, the working state determination device acquires the force sensor value and the arm tip position of the gripper of the manipulator. Then, the force sensor value and the arm tip position are input to the determiner that is coupled with the first determiner that extracts a characteristic of the normal working state and the second determiner that extracts a characteristic of the abnormal working state and determines the manipulator working state. The working state determination device determines whether the working state is work completion, work continuation, or work continuation impossible on the basis of the output value of the determiner, and controls the manipulator on the basis of the determination result. As a result, the working state including the normal system and the abnormal system can be determined.
In addition, according to the present embodiment, since the working state in which the work continuation impossible can also be determined, it is possible to promote the automation of the work by the robot system, for example, by automatically executing the operation such as regrasping the work even when the assembly work for various works is targeted. In addition, according to the present embodiment, since the user is released from feature engineering such as setting a threshold of a force sensor value for each target workpiece, it is possible to reduce a burden of starting a production line of automatic assembly.
In addition, according to the present embodiment, when the waveforms of the force sensor values in each of the normal system and the abnormal system are similar to the learning data used for learning of the determiner, the working state can be determined with high accuracy even for work on a workpiece different from the workpiece used for learning. For example, it is possible to determine the working state of the unlearned insertion and assembly work of the 9-pin type in-vehicle connector by using a determiner learned using a force sensor value at the time of the insertion and assembly work of the 4-pin type in-vehicle connector as learning data.
In addition, by making the time duration of the data input to the second determiner longer than the time duration of the data input to the first determiner, it is possible to grasp the characteristics of the data of the normal system and the abnormal system and to accurately determine the operation state.
In the embodiment, the case in which the force sensor value and the arm tip position are used as the data to be input to the determiner has been described, but the arm tip position is not essential, and at least the force sensor value may be used as an input.
In addition, when both the force sensor value and the arm tip position are used, the determination may be made by a logical product (AND) or a logical sum (OR) of the force sensor value and the arm tip position. In the first determiner that determines whether the working state is work continuation or work completion, it is desirable to perform the AND determination in order to prevent the occurrence of the defective product when the work incomplete product is erroneously determined to be completed. In the second determiner that determines whether the working state is work continuation or work continuation impossible, whether to perform the AND determination or the OR determination may be set according to the content of the work, the coping method at the time of work continuation impossible, and the like. In order to perform such an AND determination or an OR determination, it is only required to execute learning of the determiner by using learning data labeled according to the AND condition or the OR condition. In a case in which learning data labeled with respect to an AND condition of a force sensor value and an arm tip position is used, a determiner that realizes an AND determination is used, and in a case in which learning data labeled with respect to an OR condition is used, a determiner that realizes an OR determination is used.
In addition to the data actually acquired from the force sensor or the manipulator (original learning data), learning data generated in a pseudo manner using a random number may be used as the learning data. Specifically, a random number is determined from a variation element of data that can occur on site, such as sensor noise and a scale of force sensor values, and pseudo learning data is generated on the basis of the determined random number. For example, as illustrated in
Furthermore, in the embodiment, the case in which each functional configuration is implemented by one computer has been described, but the present invention is not limited thereto. For example, the working state determination device including the acquisition unit, the learning unit, and the determination unit and the control device including the control unit may be implemented by different computers. Furthermore, as for the working state determination device, a learning device including the acquisition unit and the learning unit and a determination device including the acquisition unit and the determination unit may be implemented by different computers.
In addition, the learning processing and the determination control processing executed by the CPU reading software (program) in the embodiment may be executed by various processors other than the CPU. Examples of the processor in this case include a programmable logic device (PLD) in which a circuit configuration can be changed after manufacturing a field-programmable gate array (FPGA) or the like, a dedicated electric circuit that is a processor having a circuit configuration exclusively designed for executing specific processing such as an application specific integrated circuit (ASIC), and the like. Furthermore, the learning processing and the determination control processing may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, and the like). More specifically, the hardware structure of these various processors is an electric circuit in which circuit elements such as semiconductor elements are combined.
In the embodiment, the aspect in which the working state determination program is stored (installed) in the storage device in advance has been described, but the present invention is not limited thereto. The program may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, a Blu-ray disk, or a USB memory. In addition, the program may be downloaded from an external device via a network.
The present embodiment is applicable to an industrial robot, a service robot, or the like having a function of executing automatic insertion or fitting work such as insertion of a connector or fitting of a lid component.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-081072 | May 2021 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2022/015156 | 3/28/2022 | WO |
