ASSISTANCE DEVICE, WORK MACHINE, AND ASSISTANCE SYSTEM

BACKGROUND
Technical Field

The present disclosure relates to an assistance device and the like.

Description of Related Art

For example, a technique is disclosed for generating a trajectory of a work part of a work machine when performing a predetermined movement according to a surrounding environment, a target shape, or the like of the work machine.

In the related art, a trajectory is generated for a claw tip of a bucket of an excavator when a digging movement is performed.

SUMMARY

According to one embodiment of the present disclosure, an assistance device is provided. The assistance device includes:

- an input part configured to receive an input from a user; and
- in a display part configured to determine, response to the input from the input part, specifications related to a combination of a plurality of movements of a work machine, the plurality of movements being different types from each other, and to display a first operation screen for generating a trajectory of a work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements.

According to another embodiment of the present disclosure, a work machine is provided. The work machine includes:

- an input part configured to receive an input from a user; and
- a display part configured to determine, in response to the input from the input part, specifications related to a combination of a plurality of movements of the work machine, the plurality of movements being different types from each other, and to display a first operation screen for generating a trajectory of a work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements.

According to still another embodiment of the present disclosure, an assistance system including a work machine, and an assistance device capable of communicating with the work machine is provided, wherein the work machine includes:

- an input part configured to receive an input from a user; and
- a display part configured to determine, in response to the input from the input part, specifications related to a combination of a plurality of movements of the work machine, the plurality of movements being different types from each other, and to display a first operation screen for generating a trajectory of a work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements.

According to still another embodiment of the present disclosure, a program for causing an information processing device to execute a following process is provided, wherein the information processing device includes an input part and a display part. The process including:

- determining, in response to the input from the input part, specifications related to a combination of a plurality of movements of a work machine, the plurality of movements being different types from each other; and
- causing the display part to display a first operation screen for generating a trajectory of a work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of an activation assistance system.

FIG. 2 is a top view illustrating an example of an excavator.

FIG. 3 is a diagram illustrating an example of a configuration related to remote operation of the excavator.

FIG. 4 is a block diagram illustrating an example of a hardware configuration of the excavator.

FIG. 5 is a diagram illustrating an example of a hardware configuration of a remote operation assistance device.

FIG. 6 is a functional block diagram illustrating an example of a functional configuration related to generation of a target trajectory of a work part of the excavator in the activation assistance system.

FIG. 7 is a diagram illustrating examples of a structure of a model for extracting features of a trajectory of a work part of the excavator.

FIG. 8 is a diagram illustrating examples of a structure of a model for extracting features of a trajectory of the work part of the excavator.

FIG. 9 is a diagram illustrating an example of a structure of a model for generating a trajectory of the work part in a combined movement of the excavator.

FIG. 10 is a diagram illustrating an example of a structure of a model for generating a trajectory of the work part in the combined movement of the excavator.

FIG. 11 is a diagram illustrating an example of a trajectory of the work part of the excavator in land leveling work.

FIG. 12 is a diagram illustrating an example of a screen illustrating a topographical shape around the excavator.

FIG. 13 is a diagram illustrating a first example of a setting screen for specifications related to a combination of a plurality of basic movements in a combined movement of the excavator.

FIG. 14 is a diagram illustrating a second example of a setting screen for specifications related to the combination of the plurality of basic movements in the combined movement of the excavator.

FIG. 15 is a flowchart schematically illustrating an example of a process related to generation of a trajectory of the work part of the excavator.

DETAILED DESCRIPTION

However, in the above-described technique, a trajectory of the work part can be generated only according to a basic movement (for example, a digging movement) of the work machine. Therefore, for example, when performing work by combining a plurality of movements that differ from each other, it becomes necessary to repeat the process of generating a trajectory for the work part by each of the target basic movements and moving the work part along the corresponding generated trajectory in accordance with the sequentially changing surrounding environment. As a result, work efficiency may decrease.

In view of the above-described issue, it is an object of the present disclosure to provide a technique capable of improving work efficiency when work is performed by combining a plurality of movements of a work machine.

According to the above-described embodiment, it is possible to improve work efficiency in a case where work is performed by combining a plurality of movements of the work machine.

Hereinafter, embodiments will be described with reference to the accompanying drawings.

[Overview of Activation Assistance System]

An overview of an activation assistance system SYS according to the present embodiment will be described with reference to FIGS. 1 to 3.

FIG. 1 is a diagram illustrating an example of an activation assistance system SYS. In FIG. 1, an excavator 100 is illustrated in a left side view. FIG. 2 is a top view illustrating an example of the excavator 100. FIG. 3 is a diagram illustrating an example of a configuration related to remote operation of the excavator 100. Hereinafter, a direction in the excavator 100 or a direction viewed from the excavator 100 may be described by defining a direction in which the attachment AT extends in a top view of the excavator 100 (an upper direction in FIG. 2) as “front”.

As illustrated in FIG. 1, the activation assistance system SYS includes the excavator 100 and an information processing device 200.

The activation assistance system SYS coordinates with the excavator 100 using the information processing device 200 to assist the activation of the excavator 100.

The number of excavators 100 included in the activation assistance system SYS may be one or more.

The excavator 100 is a work machine that receives assistance related to activation in the activation assistance system SYS.

The work machine included in the activation assistance system SYS to receive assistance related to activation may be another work machine different from the excavator 100. For example, the other work machine is a work machine including a work attachment, and specifically, is a crane, a forklift, a road machine, or the like. The road machine is, for example, an asphalt finisher.

As illustrated in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper slewing body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.

The lower traveling body 1 causes the excavator 100 to travel by using the crawler 1C. The crawler 1C includes a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler 1CR is hydraulically driven by the traveling hydraulic motor 1MR. Thus, the lower traveling body 1 can travel by itself.

The upper slewing body 3 is slewably mounted on the lower traveling body 1 via a slewing mechanism 2. For example, the upper slewing body 3 slews with respect to the lower traveling body 1 by the slewing mechanism 2 being hydraulically driven by the slewing hydraulic motor 2M.

The boom 4 is attached to the center of the front portion of the upper slewing body 3 so as to be able to be elevated and lowered about a rotation axis along the left-right direction. The arm 5 is attached to the distal end of the boom 4 so as to be rotatable about a rotation axis along the left-right direction. The bucket 6 is attached to the distal end of the arm 5 so as to be rotatable about a rotation axis along the left-right direction.

The bucket 6 is an example of an end attachment and is used for, for example, digging work or land leveling work.

The bucket 6 is attached to the distal end of the arm 5 in a manner that the bucket 6 can be appropriately replaced according to the work content of the excavator 100. That is, instead of the bucket 6, a bucket of a type different from the bucket 6, for example, a relatively large bucket, a slope bucket, a dredging bucket, or the like may be attached to the distal end of the arm 5. Further, an end attachment of a type other than the bucket, for example, a stirrer, a breaker, a crusher, or the like may be attached to the distal end of the arm 5. Further, for example, an auxiliary attachment such as a quick coupling or a tilt rotator may be provided between the arm 5 and the end attachment.

The boom 4, the arm 5, and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

The cabin 10 is a control room in which an operator gets into and operates the excavator 100. The cabin 10 is mounted on, for example, the left side of the front portion of the upper slewing body 3.

For example, the excavator 100 operates driven elements such as the lower traveling body 1 (that is, a pair of left and right crawlers 1CL and 1CR), the upper slewing body 3, the boom 4, the arm 5, and the bucket 6 in response to an operation of the operator who gets into in the cabin 10.

Further, the excavator 100 may be configured to be remotely operated from the outside of the excavator 100 instead of or in addition to being configured to be operable by the operator in the cabin 10. When the excavator 100 is remotely operated, the inside of the cabin 10 may be unmanned. Hereinafter, the description will be given on the assumption that the operation of the operator includes at least one of an operation by the operator in the cabin 10 on an operation device 26 and a remote operation by an external operator.

For example, as illustrated in FIG. 3, the remote operation includes a mode in which the excavator 100 is operated by an operation input related to the actuator of the excavator 100 performed by a remote operation assistance device 300. The remote operation assistance device 300 may be provided separately from the information processing device 200 or may be the information processing device 200.

The remote operation assistance device 300 is provided in, for example, a management center that manages work of the excavator 100 from the outside. The remote operation assistance device 300 may be a portable operation terminal. In this case, an operator can remotely operate the excavator 100 while directly monitoring a work situation of the excavator 100 from the surroundings of the excavator 100.

The excavator 100 may transmit image (hereinafter, referred to as a “surrounding image”) representing a surrounding situation including the front of the excavator 100 based on the captured image output by an imaging device described later to the remote operation assistance device 300 through a communication device 60 described later, for example. The excavator 100 may transmit the captured image output by the imaging device 40 to the remote operation assistance device 300 through the communication device 60, and the remote operation assistance device 300 may process the captured image received from the excavator 100 and generate a surrounding image. Then, the remote operation assistance device 300 may cause a display device to display the surrounding image representing a surrounding situation of the excavator 100 including the front of the excavator 100. Further, various information images (information screens) displayed on an output device 50 (a display device 50A) inside the cabin 10 may be similarly displayed on the display device of the remote operation assistance device 300. Thus, the operator who uses the remote operation assistance device 300 can remotely operate the excavator 100 while monitoring the display content such as the image or the information screen indicating the surrounding situation of the excavator 100 displayed on the display device. The excavator 100 may operate the actuator in response to a remote operation signal indicating a content of the remote operation received from the remote operation assistance device 300 by the communication device 60, and drive the driven elements such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6.

The remote operation may include, for example, a mode in which the excavator 100 is operated by voice input, gesture input, or the like, from the outside to the excavator 100 by a person (for example, a worker) around the excavator 100. Specifically, the excavator 100 recognizes a voice uttered by a worker, a gesture performed by the worker, or the like around the excavator 100 through a voice input device (for example, a microphone), a gesture input device (for example, an imaging device), and the like installed in the excavator 100. The excavator 100 may operate the actuators according to the recognized contents of the voice, the gesture, and the like, and drive the driven elements such as the lower traveling body 1 (the left and right crawlers 1C), the upper slewing body 3, the boom 4, the arm 5, and the bucket 6.

The work of the excavator 100 may be remotely monitored. In this case, a remote monitoring assistance device having the same function as the remote operation assistance device 300 may be provided. The remote monitoring assistance device is, for example, the information processing device 200. Thus, a monitoring person who is a user of the remote monitoring assistance device can check a situation of the work of the excavator 100 while monitoring the surrounding image displayed on the display device of the remote monitoring assistance device. Further, for example, if deemed necessary from a safety perspective, the monitoring person can intervene in the operation by the operator of the excavator 100 and bring the excavator 100 to an emergency stop by making a predetermined input using the input device of the remote monitoring assistance device.

The information processing device 200 communicates with the excavator 100 to coordinate with each other and assist the activation of the excavator 100.

The information processing device 200 is, for example, a server device or a terminal device for management installed in a management office in a work site of the excavator 100 or a management center or the like that is located at a place different from the work site of the excavator 100 and manages an activation state or the like of the excavator 100. The server device may be an on-premise server, a cloud server, or an edge server. The terminal device for management may be, for example, a stationary terminal device such as a desktop personal computer (PC) or a portable terminal device (portable terminal) such as a tablet terminal, a smartphone, or a laptop PC. In the latter case, a worker at the work site, a supervisor who supervises the work, a manager who manages the work site, or the like can move within the work site while carrying the portable information processing device 200. In the latter case, the operator can bring the portable information processing device 200 into the cabin of the excavator 100, for example.

The information processing device 200 acquires data related to the activation state from the excavator 100, for example. Thus, the information processing device 200 can identify the operation state of the excavator 100 and monitor the presence or absence of abnormality of the excavator 100. The information processing device 200 can display data related to the activation state of the excavator 100 through a display device 208 to be described later and cause the user to monitor the data.

The information processing device 200 may transmit various data such as a program and reference data used in the processes of a controller 30 or the like to the excavator 100. Thus, the excavator 100 can perform various processes related to the activation of the excavator 100 using various data downloaded from the information processing device 200.

[Hardware Configuration of Activation Assistance System]

Next, a hardware configuration of the activation assistance system SYS will be described with reference to FIGS. 4 and 5 in addition to FIGS. 1 to 3.

The hardware configuration of the remote operation assistance device 300 may be the same as that of the information processing device 200. Therefore, illustration and description of the hardware configuration of the remote operation assistance device 300 will be omitted.

FIG. 4 is a block diagram illustrating an example of a hardware configuration of the excavator 100.

In FIG. 4, a path through which mechanical power is transmitted is indicated by a double line, a path through which high-pressure hydraulic fluid for driving the hydraulic actuator flows is indicated by a solid line, a path through which pilot pressure is transmitted is indicated by a broken line, and a path through which an electric signal is transmitted is indicated by a dotted line.

The excavator 100 includes respective components such as a hydraulic drive system related to hydraulic drive of a driven element, an operation system related to operation of the driven element, a user interface system related to exchange of information with a user, a communication system related to communication with the outside, and a control system related to various controls.

<<Hydraulic Drive System>>

As illustrated in FIG. 4, the hydraulic drive system of the excavator 100 includes hydraulic actuators HA that hydraulically drive driven elements such as the lower traveling body 1 (the left and right crawlers 1C), the upper slewing body 3, and the attachment AT, as described above. The hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

The hydraulic actuators HA include traveling hydraulic motors 1ML and 1MR, a slewing hydraulic motor 2M, the boom cylinder 7, an arm cylinder 8, the bucket cylinder 9, and the like.

In the excavator 100, some or all of the hydraulic actuators HA may be replaced with electric actuators. That is, the excavator 100 may be a hybrid excavator or an electric excavator.

The engine 11 is a prime mover of the excavator 100 and is a main power source in a hydraulic drive system. The engine 11 is, for example, a diesel engine using light oil as fuel. The engine 11 is installed, for example, in a rear portion of the upper slewing body 3. The engine 11 rotates at a constant target rotation speed set in advance under direct or indirect control by the controller 30 described later, and drives the main pump 14 and a pilot pump 15.

Note that, instead of or in addition to the engine 11, another prime mover (for example, an electric motor) or the like may be installed in the excavator 100.

The regulator 13 controls (adjusts) a discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter, referred to as “tilt angle”) in response to a control command from the controller 30.

The main pump 14 supplies a hydraulic fluid to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, in the rear portion of the upper slewing body 3, similarly to the engine 11. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, the stroke length of the piston is adjusted by the tilting angle of the swash plate being adjusted by the regulator 13 under the control of the controller 30, and the discharge flow rate and the discharge pressure are controlled.

The control valve 17 drives the hydraulic actuators HA according to the content of the operation or remote operation of the operation device 26 by the operator or an operation command corresponding to an automatic driving function. The control valve 17 is mounted, for example, in a central portion of the upper slewing body 3. As described above, the control valve 17 is connected to the main pump 14 via the high-pressure hydraulic line, and selectively supplies the hydraulic fluid supplied from the main pump 14 to each hydraulic actuator in response to an operation of the operator or an operation command corresponding to the automatic driving function. Specifically, the control valve 17 includes a plurality of control valves (also referred to as “direction switching valves”) that control the flow rate and the flow direction of the hydraulic fluid supplied from the main pump 14 to each of the hydraulic actuators HA.

<<Operation System>>

As illustrated in FIG. 4, the operation system of the excavator 100 includes the pilot pump 15, the operation device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

The pilot pump 15 supplies a pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump is mounted, for example, in the rear portion of the upper slewing body 3, similarly to the engine 11. The pilot pump is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

The pilot pump 15 may be omitted. In this case, the hydraulic fluid at a relatively low pressure obtained by reducing the pressure of the hydraulic fluid at a relatively high pressure discharged from the main pump 14 by a predetermined pressure reducing valve may be supplied to various hydraulic devices as the pilot pressure.

The operation device 26 is provided near an operator's seat of the cabin 10 and is used by an operator to operate various driven elements. Specifically, the operation device 26 is used for the operator to operate the hydraulic actuators HA that drive respective driven elements, thereby implementing the operation by the operator of the driven elements to be driven by the hydraulic actuators HA. The operation device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).

For example, as illustrated in FIG. 4, the operation device 26 is a hydraulic pilot type. Specifically, the operation device 26 outputs a pilot pressure according to an operation content to a pilot line 25A on the secondary side by using the hydraulic fluid supplied from the pilot pump 15 through the pilot line 25 and a pilot line 27A branching from the pilot line 25. The pilot line 27A is connected to one of inlet ports of the shuttle valve 32, and is connected to the control valve 17 via a pilot line 27 connected to an outlet port of the shuttle valve 32. Thus, the pilot pressure according to the operation content related to various driven elements (hydraulic actuators HA) in the operation device 26 can be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operation device 26 by the operator or the like.

The operation device 26 may be an electric type operation device. In such a case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valves 33 are omitted. Specifically, the operation device 26 outputs an electric signal (hereinafter, referred to as an “operation signal”) corresponding to an operation content, and the operation signal is input to the controller 30. The controller 30 outputs, to the hydraulic control valve 31, a control command corresponding to a content of the operation signal, that is, a control signal corresponding to the operation content with respect to the operation device 26. Thus, the pilot pressure according to the operation content of the operation device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operation device 26.

Further, the control valve (direction switching valve) built in the control valve 17 for driving each hydraulic actuator HA may be an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve 17, that is, the electromagnetic solenoid type control valve.

As described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output a control command corresponding to the operation content of the operation device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or to a driver or the like that drives the electric actuator. In addition, when the excavator 100 is remotely operated, the operation device 26 may be omitted.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operation device 26 and for each driving direction (for example, the raising direction and the lowering direction of the boom 4) of the driven element (hydraulic actuator HA). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and may be configured to be able to change the flow passage areas (that is, the cross-sectional areas through which the hydraulic fluid can flow). Thus, the hydraulic control valve 31 can output a predetermined pilot pressure to the pilot line 25B on the secondary side by using the hydraulic fluid of the pilot pump 15 supplied through the pilot line 27B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure according to a control signal from the controller 30 to the control valve 17 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Therefore, for example, the controller 30 can cause the hydraulic control valve 31 to supply the pilot pressure according to the operation command corresponding to the automatic operation function to the control valve 17, and can implement the operation of the excavator 100 by the automatic operation function.

The controller 30 may control the hydraulic control valve 31 to implement remote operation of the excavator 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation designated by the remote operation signal received from the remote operation assistance device 300 to the hydraulic control valve 31 by the communication device 60. Thus, the controller 30 can cause the hydraulic control valve 31 to supply the pilot pressure according to the content of the remote operation to the control valve 17, and can implement the operation of the excavator 100 based on the remote operation by the operator.

In addition, when the operation device 26 is an electric type, the controller 30 can cause the hydraulic control valve 31 to directly supply the pilot pressure according to the operation content (operation signal) of the operation device 26 to the control valve 17, and implement the operation of the excavator 100 based on the operation of the operator.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs, to the outlet port, the hydraulic fluid having a higher pilot pressure of the pilot pressures input to the two inlet ports. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operation device 26 and for each driving direction of the driven element (hydraulic actuator HA), similarly to the hydraulic control valve 31. For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the shuttle valves 32 is connected to the pilot line 27A on the secondary side of the operation device 26 (specifically, the above-described lever device or pedal device included in the operation device 26), and the other is connected to a pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve is a control valve that drives a hydraulic actuator HA serving as an operation target of the above-described lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can cause the higher one of the pilot pressure of the pilot line 27A on the secondary side of the operation device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31 to act on the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve without depending on the operation of the operation device 26 by the operator by outputting the pilot pressure higher than the pilot pressure on the secondary side of the operation device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operations of the driven elements (the lower traveling body 1, the upper slewing body 3, and the attachment AT) independent of the operation state of the operator on the operation device 26, thereby implementing the automatic operation function and the remote operation function.

The hydraulic control valve 33 is provided in the pilot line 27A that connects the operation device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured to be able to change, for example, the flow passage area thereof. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. Thus, the controller 30 can forcibly reduce the pilot pressure output from the operation device 26 when the operation device 26 is operated by the operator. Therefore, even when the operation device 26 is operated, the controller can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operation device 26. Further, for example, even when the operation device 26 is operated, the controller 30 can reduce the pilot pressure output from the operation device 26 to be lower than the pilot pressure output from the hydraulic control valve 31. Therefore, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, independent of the operation content of the operation device 26 by controlling the hydraulic control valve 31 and the hydraulic control valve 33. Therefore, the controller 30 can more appropriately implement the automatic operation function and the remote operation function of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

<<User Interface System>>

As illustrated in FIG. 4, the user interface system of the excavator 100 includes the operation device 26, the output device 50, and an input device 52.

The output device 50 outputs various kinds of information to a user of the excavator 100 (for example, an operator of the cabin 10 or an operator of an external remote operation), a person around the excavator 100 (for example, a worker or an operator of an operation vehicle), or the like.

For example, the output device 50 includes an illumination device, the display device 50A (see FIG. 6), or the like that outputs various kinds of information in a visual manner. The illumination device is, for example, a warning lamp (indicator lamp) or the like. The display device 50A is, for example, a liquid-crystal display, an organic electroluminescence (EL) display, or the like. For example, as illustrated in FIG. 2, the illumination device and the display device 50A may be provided inside the cabin 10, and output various kinds of information to an operator or the like inside the cabin 10 in a visual manner. The illumination device and the display device 50A may be provided on, for example, a side surface of the upper slewing body 3, and may output various kinds of information to an operator or the like around the excavator 100 in a visual manner.

The output device 50 may include a sound output device that outputs various kinds of information by an auditory method. The sound output device includes, for example, a buzzer, a speaker, and the like. The sound output device may be provided, for example, on at least one of the inside and the outside of the cabin 10, and output various kinds of information to the operator inside the cabin 10 or a person (worker or the like) around the excavator 100 by an auditory method.

The output device 50 may include a device that outputs various kinds of information by a tactile method such as vibration of the operator's seat.

The input device 52 receives various inputs from the user of the excavator 100, and signals corresponding to the received inputs are incorporated into the controller 30. For example, as illustrated in FIG. 2, the input device 52 is provided inside the cabin 10 to receive an input from an operator or the like inside the cabin 10. The input device 52 may be provided on, for example, a side surface of the upper slewing body 3 to receive an input from a worker or the like around the excavator 100.

For example, the input device 52 includes an operation input device that receives an input by a mechanical operation from a user. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided in the operation device 26 (lever device), and the like.

The input device 52 may include a voice input device that receives a voice input from a user. The voice input device includes, for example, a microphone.

The input device 52 may include a gesture input device that receives a gesture input of a user. The gesture input device includes, for example, an imaging device that images a state of a gesture performed by the user.

The input device 52 may include a biometric input device that receives a biometric input of a user. The biometric input includes, for example, input of biometric information such as a fingerprint or an iris of the user.

<<Communication System>>

As illustrated in FIG. 4, the communication system of the excavator 100 according to the present embodiment includes the communication device 60.

The communication device 60 is connected to an external communication line and communicates with a device provided from the excavator 100. The device provided separately from the excavator 100 may include a portable terminal device (portable terminal) brought into the cabin 10 by the user of the excavator 100, in addition to the device outside the excavator 100. The communication device 60 may include, for example, a mobile communication module conforming to a standard such as 4G (4th Generation) or 5G (5th Generation). The communication device 60 may include, for example, a satellite communication module. The communication device 60 may include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. The communication device 60 may include a plurality of communication devices according to communication lines to be connected.

For example, the communication device 60 communicates with an external device such as the information processing device 200 or the remote operation assistance device 300 within a work site through a local communication line constructed in the work site. The local communication line is, for example, a mobile communication line of a localized 5th Generation (so-called local 5G) constructed at the work site or a local network (local area network: LAN) of a WiFi6.

The communication device 60 may communicate with the information processing device 200, the remote operation assistance device 300, and the like outside the work site through a communication line of a wide area including the work site, that is, a wide area network (WAN). The wide area network includes, for example, a wide area mobile communication network, a satellite communication network, the Internet, and the like.

<<Control System>>

As illustrated in FIG. 4, the control system of the excavator 100 includes the controller 30. The control system of the excavator 100 according to the present embodiment includes an operation pressure sensor 29, the imaging device 40, and sensors S1 to S5.

The controller 30 performs various controls related to the excavator 100.

The functions of the controller 30 may be implemented by any given hardware, or a combination of any given hardware and software, or the like. For example, as illustrated in FIG. 3, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a central processing unit (CPU) 30C, and an interface device 30D, which are connected to each other via a bus BS1.

The auxiliary storage device 30A is a non-volatile storage part, and stores a program to be installed as well as storing necessary files and data. The auxiliary storage device 30A is, for example, an electrically erasable programmable read-only memory (EEPROM), a flash memory, or the like.

The memory device 30B loads the program in the auxiliary storage device 30A such that a CPU (Central Processing Unit) 30C can read the program, for example, when an instruction to start the program is given. The memory device 30B is, for example, a static random access memory (SRAM).

The CPU 30C executes, for example, a program loaded into the memory device 30B, and implements various functions of the controller 30 according to instructions of the program. The interface device 30D functions as, for example, a communication interface for connection to a communication line inside the excavator 100. The interface device 30D may include a plurality of different types of communication interfaces according to the type of communication line to be connected.

The interface device 30D functions as an external interface for reading and writing of information from and to a recording medium. The recording medium is, for example, a dedicated tool that is connected to a connector installed inside the cabin 10 by a detachable cable. The recording medium may be a general-purpose recording medium such as a secure digital (SD) memory card or a universal serial bus (USB) memory. Thus, the program for implementing various functions of the controller 30 can be provided by, for example, a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The program may be downloaded from another computer outside the excavator 100 through the communication device 60 and installed in the auxiliary storage device 30A.

Note that some of the functions of the controller may be implemented by another controller (control device). That is, the functions of the controller 30 may be implemented by a plurality of controllers in a distributed manner.

The operation pressure sensor 29 detects a pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot-type operation device 26, that is, a pilot pressure according to the operation state of each of the driven elements (hydraulic actuators) in the operation device 26. A detection signal of the pilot pressure according to the operation state of each driven element (hydraulic actuator HA) in the operation device 26 by the operation pressure sensor 29 is incorporated into the controller 30.

When the operation device 26 is an electric type, the operation pressure sensor 29 is omitted. This is because the controller 30 can identify the operation state of each driven element through the operation device 26 based on the operation signal taken in from the operation device 26.

The imaging device 40 acquires images around the excavator 100. The imaging device 40 may acquire (generate) three dimensional data (hereinafter, simply referred to as “three dimensional data of an object”) indicating the position and the outer shape of an object around the excavator 100 in the imaging range (angle of view) based on the acquired image and data related to a distance described later. The three dimensional data of the object around the excavator 100 is, for example, data of coordinate information of a point group representing the surface of the object, distance image data, or the like.

For example, as illustrated in FIG. 2, the imaging device 40 includes a camera 40F that images the front side of the upper slewing body 3, a camera 40B that images the back side of the upper slewing body 3, a camera 40L that images the left side of the upper slewing body 3, and a camera 40R that images the right side of the upper slewing body 3. Thus, the imaging device 40 can image the entire circumference around the excavator 100, that is, a range over an angular direction of 360 degrees in a top view of the excavator 100. The operator can visually recognize the captured images of the cameras 40B, 40L, and 40R, and the surrounding images such as the processed images generated based on the captured images through the display device 50A and the remote operation assistance device 300, and can check the states of the left side, the right side, and the back side of the upper slewing body 3. Further, the operator can remotely operate the excavator 100 while monitoring the operation of the attachment AT by visually recognizing the captured image of the camera 40F, and the surrounding image such as the processed image generated based on the captured image through the remote operation assistance device 300. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as a “camera 40X”.

The camera 40X is, for example, a monocular camera. The camera 40X may be capable of acquiring depth information in addition to a two dimensional image, such as a stereo camera or a TOF (Time Of Flight) camera, or the like (hereinafter collectively referred to as a “3D camera”).

The controller 30 receives the image captured by the imaging device 40 (camera 40X) via a one-to-one communication line or an in-vehicle network. Thus, for example, the controller 30 can check an object around the excavator 100 based on the camera 40X. Further, for example, the controller 30 can determine the surrounding environment of the excavator 100 based on the camera 40X. In addition, for example, the controller 30 can determine the attitude state of the attachment AT in the captured image based on the camera 40X (camera 40F). Further, for example, the controller 30 can determine the attitude state of a body (the upper slewing body 3) of the excavator 100 with reference to an object around the excavator 100.

Note that some or all of the cameras 40B, 40L, and 40R may be omitted. Instead of or in addition to the imaging device 40 (camera 40X), a distance sensor may be provided in the upper slewing body 3. The distance sensor is attached to, for example, an upper portion of the upper slewing body 3, and acquires data related to the distance and direction of a surrounding object with respect to the excavator 100. The distance sensor may acquire (generate) three dimensional data (for example, data of coordinate information of a point group) of an object around the excavator 100 in the sensing range based on the acquired data. The distance sensor is, for example, a light detection and ranging (LIDAR). Further, for example, the distance sensor may be a millimeter wave radar, an ultrasonic sensor, an infrared sensor, or the like. The sensor S1 is attached to the boom 4 and detects an attitude angle (hereinafter, referred to as a “boom angle”) around a rotation axis of a base end corresponding to a coupling portion of the boom 4 with the upper slewing body 3. The sensor S1 includes, rotary potentiometers, rotary encoders, accelerometers, angular accelerometers, six axis sensors, and IMU (Inertial Measurement Unit). Hereinafter, the same may be applied to the sensors S2 to S4. The sensor S1 may include a cylinder sensor that detects the extension/contraction position of the boom cylinder 7. The same applies to the sensors S2 and S3. A detection signal of the boom angle by the sensor S1 is incorporated into the controller 30. Thus, the controller 30 can identify the attitude state of the boom 4.

The sensor S2 is attached to the arm 5 and detects an attitude angle (hereinafter, referred to as an “arm angle”) around a rotation axis of a base end corresponding to a coupling portion of the arm 5 with the boom 4. A detection signal of the arm angle by the sensor S2 is incorporated into the controller 30. Thus, the controller 30 can identify the attitude state of the arm 5.

The sensor S3 is attached to the bucket 6 and detects an attitude angle (hereinafter, referred to as an “arm angle”) around a rotation axis of a base end corresponding to a coupling portion of the bucket 6 with the arm 5. A detection signal of the arm angle by the sensor S3 is incorporated into the controller 30. Thus, the controller can identify the attitude state of the bucket 6.

The sensor S4 detects an inclined state of the body (for example, the upper slewing body 3) with respect to a predetermined reference surface (for example, a horizontal plane). The sensor S4 is attached to, for example, the upper slewing body 3, and detects inclination angles (hereinafter, referred to as a “front-back inclination angle” and a “left-right inclination angle”) around two axes in the front-back direction and the left-right direction of the excavator 100 (that is, the upper slewing body 3). Detection signals corresponding to the inclination angles (the front-back inclination angle and the left-right inclination angle) detected by the sensor S4 are incorporated into the controller 30. Thus, the controller 30 can identify the inclination state of the body (upper slewing body 3).

The sensor S5 is attached to the upper slewing body 3 and outputs detection information related to a slewing state of the upper slewing body 3. The sensor S5 detects, for example, a slewing angular speed and a slewing angle of the upper slewing body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, and the like. Detection information related to the slewing state detected by the sensor S5 is incorporated into the controller 30. Thus, the controller 30 can identify the slewing state such as the slewing angle of the upper slewing body 3.

Note that, in a case where the sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like capable of detecting angular velocities about three axes, the slewing state (for example, slewing angular velocities) of the upper slewing body 3 may be detected based on detection signals of the sensor S4. In this case, the sensor S5 may be omitted. In addition, when it is possible to identify the attitude states of the upper slewing body 3, the attachment AT, or the like based on the output of the imaging device 40 or the distance sensor, at least some of the sensors S1 to S5 may be omitted.

<<Hardware Configuration of Information processing Device>

FIG. 5 is a block diagram illustrating an example of a hardware configuration of the information processing device 200.

The functions of the information processing device 200 are implemented by any given hardware, a combination of any given hardware and software, or the like. For example, as illustrated in FIG. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU (Central Processing Unit) 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, the display device 208, and a sound output device 209, which are connected by a bus BS2.

The external interface 201 functions as an interface for reading from recording media 201A and writing to the recording media 201A. The recording media 201A include, for example, flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BD (Blu-ray (trademark) Disc), SD memory cards, USB (Universal Serial Bus) memories, and the like. The information processing device 200 can read various kinds of information used in processing through the recording media 201A, store the information in the auxiliary storage device 202, and install programs for implementing various functions.

The information processing device 200 may acquire various data and programs used in processing from an external device via the communication interface 206.

The auxiliary storage device 202 stores the installed various programs, and also stores files, data, and the like necessary for various processes. The auxiliary storage device 202 includes, for example, a hard disc drive (HDD), an SSD (Solid State Disc), a flash memory, or the like.

When an instruction to activate a program is issued, the memory device 203 reads the program from the auxiliary storage device 202 and stores the program. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) or an SRAM.

The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203 and implements various functions related to the information processing device 200 according to the programs.

The high-speed arithmetic device 205 performs arithmetic processing at a relatively high speed in conjunction with the CPU 204. The high-speed arithmetic device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a FPGA (Field-Programmable Gate Array), or the like.

The high-speed arithmetic device 205 may be omitted depending on the speed of necessary arithmetic processing.

The communication interface 206 is used as an interface for connecting to an external device so as to be able to communicate with the external device. Thus, the information processing device 200 can communicate with an external device such as the excavator 100 through the communication interface 206. The communication interface 206 may include a plurality of types of communication interfaces depending on a communication system with a device to be connected.

The input device 207 receives various inputs from a user. The input device 207 includes a remote operation device for performing remote operation of the excavator 100. The input device 207 includes, for example, an input device (hereinafter, referred to as an “operation input device”) in a form of receiving a mechanical operation input from a user. The operation device for remote operation may be an operation input device. The operation input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, and the like.

The input device 207 may include a voice input device capable of receiving a voice input from the user. The voice input device includes, for example, a microphone capable of collecting a voice of the user.

The input device 207 may include a gesture input device capable of receiving a gesture input from the user. The gesture input device includes, for example, a camera capable of capturing an image of a gesture of the user.

The input device 207 may include a biometric input device capable of receiving a biometric input from the user. The biometric input device includes, for example, a camera capable of acquiring image data containing information on a fingerprint or an iris of the user.

The display device 208 displays an information screen and an operation screen to a user of the information processing device 200. The display device 208 is, for example, a liquid crystal display or an organic electroluminescence (EL) display.

The sound output device 209 transmits various kinds of information to the user of the information processing device 200 by sound. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[Functional Configuration for Generation of Trajectory of Work Part]

Next, a functional configuration related to generation of a trajectory of a work part of the excavator 100 in the activation assistance system SYS will be described with reference to FIGS. 6 to 11 in addition to FIGS. 1 to 5. FIG. 6 is a functional block diagram illustrating an example of a functional configuration related to generation of a trajectory of a work part of the excavator 100 in the activation assistance system SYS. FIGS. 7 and 8 are diagrams illustrating examples of structures of models for extracting features of trajectories of a work part of the excavator 100. FIGS. 9 and 10 are diagrams illustrating an example of a structure of a model for generating a trajectory of the work part in a combined movement of the excavator 100. FIG. 11 is a diagram illustrating an example of a trajectory of a work part of the excavator 100 in the land leveling work.

Hereinafter, the term “trajectory” is used to include both a path (that is, a locus) along which the work part of the excavator 100 has already moved and a path along which the work part of the excavator 100 may move in the future. In addition, a model for extracting features of a trajectory of the work part of the excavator 100 in a case where the model is in any learning process of before learning, during learning, and after learning is referred to as a “training model M2”, and in a case where the model is after learning is referred to as a “trained model LM2”. The work part of the excavator 100 is, for example, the claw tip or the back surface of the bucket 6. In addition, a model for generating a trajectory of the work part of the excavator 100 in a combined movement in a case where the model is in any learning process of before learning, during learning, and after learning is referred to as a “training model M3”, and in a case where the model is after learning is referred to as a “trained model LM3”. The combined movement of the excavator 100 is a movement obtained by combining two or more predetermined movements (hereinafter, referred to as “basic movements”) of different types from each other of the excavator 100.

The excavator 100 includes an assistance device 150. The assistance device 150 assists the work of the excavator 100.

As illustrated in FIG. 6, the assistance device 150 includes the operation device 26, the controller 30, the imaging device 40, the output device 50, and the input device 52. In addition, in a case where the excavator 100 is remotely operated, the assistance device 150 may include the communication device 60.

The controller 30 includes a movement log providing part 301 and a work assistance part 302 as functional parts.

In a case where the activation assistance system SYS includes a plurality of excavators 100, the controller may include an excavator 100 including only the movement log providing part 301, and an excavator 100 including only the work assistance part 302. In this case, the former excavator 100 has only a function of acquiring the movement log of the excavator 100 and providing the movement log to the information processing device 200, which is used for the work assistance function of the latter excavator 100.

The information processing device 200 includes a movement log acquiring part 2001, a movement log storage part 2002, a training data generating part 2003, a machine learning part 2004, a trained model storage part 2005, and a distribution part 2006, as functional parts.

The movement log providing part 301 is a functional part for acquiring a movement log of the excavator 100, which is original data for implementing a function of generating a trajectory of a work part of the excavator 100, and for providing the movement log to the information processing device 200. Specifically, the movement log acquiring part 301 acquires a movement log when an operator (hereinafter, referred to as a “skilled operator” for convenience) who has been operating the excavator 100 for a long time and has relative experience, and provides the movement log to the information processing device 200.

The movement log of the excavator 100 includes data related to the shape of a work target around the excavator 100 and data related to the movement of the excavator 100 performed on the shape of the work target. The data related to the shape of the work target around the excavator 100 is, for example, data related to the topographical shape of the ground at the work site as the work target of the excavator 100. The data related to the shape of the work target of the excavator 100 is, for example, image data of the imaging device 40 or three dimensional data of the work target obtained from the image data. The data related to the movement of the excavator 100 is, for example, data representing the operation content of the operator. The data representing the operation content of the operator is, for example, output data of the operation pressure sensor 29 corresponding to the hydraulic pilot type operation device 26 or output data (data of an operation signal) of the operation device 26 corresponding to the electric operation device 26. The data related to the movement of the excavator 100 may be data representing the movement state of the excavator 100 actually executed in response to the operation of the operator. The data indicating the movement state of the excavator 100 is, for example, data output from the sensors S1 to S5 or data related to the attitude state of the excavator 100 acquired from the data output from the sensors S1 to S5.

The movement log providing part 301 includes a movement log recording part 301A, a movement log storage part 301B, and a movement log transmission part 301C.

The movement log recording part 301A acquires a movement log of the excavator 100 and records the movement log in the movement log storage part 301B. For example, every time the movement of the excavator 100 is executed, the movement log recording part 301A records, in the movement log storage part 301B, data related to the shape of the work target around the excavator 100 at the start of execution of the movement or immediately before the execution of the movement of the excavator 100.

The movement log storage part 301B stores movement logs of the excavator 100 in an accumulated manner. For example, in the movement log storage part 301B, data related to the shape of the work target around the excavator 100 and data related to the movement of the excavator 100 are stored in association with each other for each movement of the excavator 100. For example, the movement log storage part may record representing 301B accumulate data a correspondence between data related to the shape of the work target around the excavator 100 and data related to the movement of the excavator 100 for each movement of the excavator 100 to construct a database for the movement logs.

The movement logs in the movement log storage part 301B, which have been transmitted to the information processing device 200 by the movement log transmission part 301C described later, may be deleted later.

The movement log transmission part 301C transmits the movement logs of the excavator 100 stored in the movement log storage part 301B to the information processing device 200 through the communication device 60. The movement log transmission part 301C may transmit record data representing a correspondence between the movement of the excavator 100 and the shape of the work target around the excavator 100 for each movement of the excavator 100 to the information processing device 200.

For example, the movement log transmission part 301C transmits a movement log of the excavator 100 which is stored in the movement log storage part 301B and has not been transmitted to the information processing device 200 in response to a signal (hereinafter, referred to as a “transmission request signal”) for requesting the transmission of the movement log of the excavator 100 from the information processing device 200. The movement log transmission part 301C may also automatically transmit the movement log of the excavator 100 which is stored in the movement log storage part 301B and has not been transmitted to the information processing device 200 at a predetermined timing. The predetermined timing is, for example, the time of stopping the activation of the excavator 100 (the time of turning off the key switch), the time of starting the activation (the time of turning on the key switch), or the like.

The movement log acquiring part 2001 acquires the movement log of the excavator 100 received from the excavator 100.

The movement log acquiring part 2001 acquires the movement log of the excavator 100 by transmitting a transmission request signal to the excavator 100 in response to an operation of the user of the information processing device 200 or automatically at a predetermined timing. The movement log acquiring part 2001 may acquire the movement log of the excavator 100 automatically transmitted from the excavator 100 at a predetermined timing.

The movement log storage part 2002 stores the movement logs of the excavator 100 acquired by the movement log acquiring part 2001 in an accumulated manner. For example, in the movement log storage part 2002, as in the case of the movement log storage part 301B, data related to the shape of the work target around the excavator 100 and the movement of the excavator 100 are stored in association with each other for each movement of the excavator 100.

The training data generating part 2003 generates training data for machine learning based on the movement logs of the excavator 100 in the movement log storage part 2002, and outputs a training data set which is an aggregate of a large number of training data. The training data generating part 2003 may automatically generate training data by batch processing or may generate training data in response to an input from the user of the information processing device 200. The training data generating part 2003 includes training data generating parts 2003A to 2003C.

The training data generating part 2003A generates training data for generating a trained model LM1 described later. For example, the training data generating part 2003A generates training data which is a combination of input data related to the shape of the work target around the excavator 100 and ground truth output data corresponding to the input data, which represents a trajectory (a locus) of the work part of the excavator 100 by the operation of the skilled operator.

The training data generating part 2003B generates training data for generating a trained model LM2. For example, the training data generating part 2003B generates training data that is a combination of input data and ground truth output data with the same data representing the trajectory (locus) of the work part in a predetermined movement (basic movement) of the excavator 100 operated by the skilled operator. In this case, the training data generating part 2003B may generate training data on a per plurality of basic movements basis where the plurality of basic movements are different types from each other (hereinafter simply referred to as “plurality of basic movements”) and output a training data set on a per plurality of basic movements basis. The training data generating part 2003B may generate training data independent of the different types of the plurality of basic movements and output a training data set that contains a mixture of training data corresponding to the plurality of basic movements.

The plurality of basic movements of the excavator 100 include, for example, at least two of a sweeping-out movement, a horizontal-pulling movement, a rolling-compaction movement, a broom-turning movement, a digging movement, an earth removal movement, and the like, which are used in the land leveling work. The sweeping-out movement is, for example, moving the attachment AT to push the bucket 6 forward along the ground and sweeping out the earth forward with the back surface of the bucket 6. In the sweeping-out movement, for example, the attachment AT performs a lowering movement of the boom 4 and an opening movement of the arm 5. The horizontal-pulling movement is, for example, moving the attachment AT to move the claw tip of the bucket 6 so as to pull the claw tip substantially horizontally toward the front along the ground and leveling the unevenness of the ground (the surface of the terrain). In the horizontal-pulling movement, for example, the attachment AT performs the movement of raising the boom 4 and the movement of closing the arm 5. The rolling-compaction movement is, for example, moving the attachment AT to press the ground with the back surface of the bucket 6. The rolling-compaction movement may be pressing the ground by hitting the back surface of the bucket 6 against the ground while moving the bucket 6 up and down. The rolling-compaction movement may be pushing the bucket 6 forward along the ground to sweep out the earth to a predetermined position in front of the bucket 6 with the back surface of the bucket 6 and then pressing the ground at the predetermined position with the back surface of the bucket 6. In the rolling-compaction movement, for example, the attachment AT performs a lowering movement of the boom 4 when pressing the ground. The broom-turning movement is, for example, moving the upper slewing body 3 to slew the bucket 6 to the left and right with the bucket being along the ground. The broom-turning movement may be, for example, moving the attachment AT and the upper slewing body 3 to push the bucket 6 forward while slewing the bucket 6 alternately to the left and right in a state with the bucket 6 being along the ground. In the broom-turning movement, for example, the upper slewing body 3 repeats an alternately slewing movement to the left and right. In the broom-turning movement, for example, the attachment AT may perform the lowering movement of the boom 4 and the opening movement of the arm 5 in addition to alternately slewing movement of the upper slewing body 3 to the left and right, similarly to the sweeping-out movement. The digging movement is, for example, moving the attachment AT to dig the earth at a certain place on the ground and scoop the earth into the bucket 6. The earth removal movement is moving the attachment AT to remove the earth scooped up by the bucket 6 in the digging movement to another place on the ground. In the earth removal movement, the excavator 100 may move the upper slewing body 3 in addition to the attachment AT. The plurality of basic movements of the excavator 100 include, for example, at least two of a digging movement, a boom raising-slewing movement, a boom lowering-slewing movement, the earth removal movement, the broom-turning movement, and the like, which are used in the digging work. The plurality of basic movements of the excavator 100 include, for example, an earth cutting movement, a rolling-compaction movement, and the like used in the slope construction work.

The training data generating part 2003C generates training data for generating the trained model LM3. For example, the training data generating part 2003C generates a trajectory (a locus) of the work part of the excavator 100 operated by the skilled operator. The training data set output by the training data generating part 2003C may include trajectories of the work part of the excavator 100 in the basic movements of different types from each other. The training data set output by the training data generating part 2003C may include a trajectory of the work part in the combined movement of the excavator 100 instead of or in addition to the trajectories of the work part in the basic movements of the excavator 100.

The trajectory (locus) of the work part of the excavator 100 is generated based on, for example, outputs from the sensors S1 to S5 included in data related to the movements of the excavator 100.

The machine learning part 2004 causes a basic training model to perform machine learning based on the set of training data generated by the training data generating part 2003, and generates a trained model. The trained model (basic training model) includes, for example, a neural network such as a deep neural network (DNN).

The machine learning part 2004 includes machine learning parts 2004A to 2004C.

The machine learning part 2004A causes the basic training model M1 to perform machine learning based on the training data set output from the training data generating part 2003A. Thus, the machine learning part 2004A can generate the trained model LM1 capable of generating (outputting) the trajectory of the work part in the basic movements of the excavator 100. Specifically, the machine learning part 2004A generates the trained model LM1 on a per plurality of basic movements basis, based on the training data set on a per plurality of basic movements basis output from the training data generating part 2003A.

The machine learning part 2004B causes the basic training model M2 to perform machine learning based on the training data set output from the training data generating part 2003B, and generates the trained model LM2. As described above, the trained model LM2 is a model for extracting the features of a trajectory of the work part of the excavator 100.

For example, as illustrated in FIG. 7, the training model M2 is a neural network that receives data representing a trajectory of the work part of the excavator 100, repeats downsampling, then repeats upsampling, and outputs data representing the trajectory of the work part of the excavator 100. In the first half of the training model M2, sequential downsampling is used to generate data with the same dimension as the input data (latent variable) for a training model M3 described later. In the second half of the trained model LM2, sequential upsampling is used to output data on the trajectory of the work part of the excavator 100.

The machine learning part 2004B causes the training model M2 to perform machine learning based on a training data set in which input data and output data are a combination of data representing the same trajectory of the work part of the excavator 100. This enables the machine learning part 2004B to generate, using the data representing the trajectory of the work part of the excavator 100 as input data, the trained model LM2 capable of outputting the same output data as the input data. In this case, the machine learning part 2004B may cause the training model M2 to perform machine learning such that the upsampling process of the second half of the training model M2 reflects the downsampling results of the corresponding first half. This enables the machine learning part 2004B to cause the training model M2 to perform machine learning such that the training model M2 can properly output the same output data as the input data.

As described above, the upsampling process (the second half) of the training model M2 reflects features for generating the same data as the input data from the intermediate data in dimensions corresponding to the latent variable, that is, the features for generating the data representing the trajectory of the work part of the excavator 100 from the intermediate data. Therefore, various network parameters in the second half of the trained model LM2 can be extracted as the features of the trajectory of the work part of the excavator 100 corresponding to the input data of the trained model LM2.

For example, as illustrated in FIG. 8, in this example, the trained model LM2 can be used to extract network parameters A1 to A4 at respective layers of the second half of the trained model LM2 as a feature F_A of a trajectory of the work part in the sweeping-out movement of the excavator 100. Similarly, in the present example, the trained model LM2 can be used to extract network parameters B1 to B4 at respective layers of the second half of the trained model LM2 as a feature F B of a trajectory of the work part in the horizontal-pulling movement of the excavator 100.

The trained model LM2 may be generated on a per plurality of basic movements basis, or one common trained model LM2 may be generated between the plurality of basic movements.

The machine learning part 2004C causes the basic training model M3 to perform machine learning, based on the training data set output from the training data generating part 2003C, and generates a trained model LM3. As described above, the trained model LM3 is a model for generating a trajectory of the work part in the combined movement of the excavator 100.

For example, as illustrated in FIG. 9, the training model M3 is a neural network that repeats upsampling using a latent variable z as input data and generates data in dimensions corresponding to a trajectory of the work part of the excavator 100. The trained model LM3 corresponds to a generator in a generative adversarial network (GAN). Then, the machine learning part 2004C causes the generator (training model M3) and a discriminator to perform learning adversarially using the training data output from the training data generating part 2003C. Specifically, the discriminator performs learning so as to be able to distinguish data representing the trajectory of the work part of the excavator 100 by the skilled operator, which is input as training data, as “real” data and data representing the trajectory of the work part of the excavator 100 generated by the generator as “fake” data. The generator performs learning so as to be able to generate data representing the trajectory of the work part of the excavator 100 that cannot be discriminated by the discriminator from the training data (data representing the trajectory of the work part of the excavator 100 by the skilled operator). The machine learning part 2004C alternately causes the discriminator and the generator to perform learning, and thus can generate the trained model LM3 capable of generating the trajectory of the work part of the excavator 100 from the latent variable z.

Here, as illustrated in FIGS. 7 to 10, as described above, the structure of the second half of the trained model LM2 (training model M2) and the structure of the trained model LM3 (training model M3) are the same. Therefore, the trained model LM3 can generate the trajectory of the work part of the excavator 100 corresponding to the features by reflecting the features extracted from the trained model LM2 of FIG. 8 in the trained model LM3. Therefore, the trained model LM3 can generate the trajectory of the work part in the combined movement of the excavator 100 by appropriately reflecting the mixed features of the basic movements different from each other, which are extracted from the trained model LM2.

For example, as illustrated in FIG. 10, in the present example, a feature FV obtained by combining the features F_A and F_B of respective trajectories of the work part in the sweeping-out movement and the horizontal-pulling movement of the excavator 100 is reflected in the trained model LM3. Specifically, the feature FV includes the features FV1 to FV4 corresponding to the network parameters A1 and B1, the network parameters A2 and B2, the network parameters A3 and B3, and network parameters A4 and B4, respectively. The feature FV1 is adjusted in a range between the network parameter A1 and the network parameter B1 by combining the network parameters A1 and B1 corresponding to the same layer of the network, and is input to a target layer of the trained model LM3. Similarly, the feature FV2 is adjusted in a range between the network parameter A2 and the network parameter B2 by combining the network parameters A2 and B2 corresponding to the same layer of the network, and is input to a target layer of the trained model LM3. Similarly, the feature FV3 is adjusted in a range between the network parameter A3 and the network parameter B3 by combining the network parameters A3 and B3 corresponding to the same layer of the network, and is input to a target layer of the trained model LM3. Similarly, the feature FV4 is adjusted in a range between the network parameter A4 and the network parameter B4 by combining the network parameters A4 and B4 corresponding to the same layer of the network, and is input to a target layer of the trained model LM3. Thus, the trained model LM3 can generate the trajectory of the work part in the combined movement of the combination of the sweeping-out movement and the horizontal-pulling movement of the excavator 100. In this example, the trained model LM3 reflects (inputs) the adjustment parameter in each layer in addition to the feature FV, and thus can adjust details of the trajectory of the work part in the combined movement of the combination of the sweeping-out movement and the horizontal-pulling movement of the excavator 100 (see the solid line arrow in the figure). Further, the adjustment parameters may be reflected in the output of the trained model LM3, such that the combined trajectory of the excavator 100 itself, as output from the trained model LM3, can be adjusted (see a broken line arrow in the figure).

For example, as illustrated in FIG. 11, there is a case where earth piles 1102 and 1103 are present in front of and behind a depression 1101 in the ground as viewed from the excavator 100 (upper slewing body 3). In this case, the excavator 100 can move the earth in the earth piles 1102 and 1103 to the depression 1101 at once and level the ground flat by using the trajectory 1100 of the work part in the combined movement, which is a combination of the sweeping-out movement and the horizontal-pulling movement of the excavator 100.

Returning to FIG. 6, the trained models LM1 to LM3 output by the machine learning part 2004 are stored in the trained model storage part 2005. In addition, in a case where the machine learning part 2004A causes the trained model LM1 to perform relearning or additional learning, the trained model LM1 in the trained model storage part 2005 is updated. The same applies to a case where the trained models LM2 and LM3 are caused to perform relearning or additional learning by the machine learning parts 2004B and 2004C.

The distribution part 2006 distributes the trained models LM1 to LM3 to the excavator 100.

For example, when the trained model LM1 is generated or updated by the machine learning part 2004A, the distribution part 2006 distributes the trained model LM1 generated or updated most recently to the excavator 100. The distribution part 2006 may distribute the latest trained model LM1 in the trained model storage part 2005 to the excavator 100, in response to a signal for requesting distribution of the trained model LM1 received from the excavator 100. The same may apply to the trained models LM2 and LM3.

The work assistance part 302 is a functional part for assisting work of the excavator 100 by an operation of the operator.

The work assistance part 302 includes a trained model storage part 302A, a work target shape acquiring part 302B, a work selecting part 302C, a trajectory generating part 302D, a specifications setting part 302E, a trajectory generating part 302F, a movement control part 302G, and a display processing part 302H.

The trained models LM1 to LM3 distributed from the information processing device 200 and received through the communication device 60 are stored in the trained model storage part 302A.

The work target shape acquiring part 302B acquires a shape (a topographical shape) of a work target around the excavator 100 based on outputs of the imaging device 40 and the distance sensor.

The work selecting part 302C selects work to be performed by the excavator 100 from among a plurality of work candidates of different types from each other, in response to an input from the user (operator) received through the input device 52. Thus, the user can generate a trajectory of the work part of the excavator 100 according to a type of work performed by the excavator 100. The plurality of work candidates include, for example, a land leveling work, a digging work, a slope construction work, and the like. In addition, in a case where the excavator 100 is remotely operated, the work selecting part 302C may select work to be performed by the excavator 100 from among a plurality of work candidates, in response to an input from the user who uses the remote operation assistance device 300, which is received through the communication device 60. For each work to be selected by the work selecting part 302C, basic movements of the excavator 100 that can be combined as a combined movement are defined in advance. For example, as described above, in the case of the land leveling work, the basic movements that can be combined as a combined movement are the broom-turning movement, the horizontal-pulling movement, the rolling-compaction movement, the digging movement, the earth removal movement, and the like. Therefore, the work selecting part 302C selects the basic movements to be combined as the combined movement of the excavator 100 by selecting the type of work.

The trajectory generating part 302D generates a trajectory of the work part of the excavator 100 on a per plurality of basic movements basis corresponding to the work selected by the work selecting part 302C, based on data related to the shape of the work target around the excavator 100, using the trained model LM1. The plurality of basic movements corresponding to each of a plurality of candidate works are defined in advance as described above. For example, the plurality of basic movements corresponding to the land leveling work include, for example, a sweeping-out movement, a horizontal-pulling movement, and a rolling-compaction movement. The plurality of basic movements corresponding to the land leveling work may include a digging movement, an earth removal movement, a broom-turning movement, and the like.

The specifications setting part 302E sets (adjusts) specifications related to a combination of a plurality of basic movements in the combined movement of the excavator 100, in response to an input from a user (operator) received through the input device 52. In addition, in a case where the excavator is 100 remotely operated, the specifications setting part 302E may set the specifications related to the combination of the plurality of basic movements of the excavator 100 in response to an input from a user who uses the remote operation assistance device 300, which is received through the communication device 60.

For example, the specifications setting part 302E sets (selects) basic movements to be combined as a combined movement of the excavator 100 from among a plurality of basic movements defined for the work selected by the work selecting part 302C. The specifications setting part 302E may adjust a composition ratio (hereinafter, “combination ratio”) on a per plurality of basic movements basis in the combined movement, with the entire combined movement of the excavator 100 being 100%. Thus, when generating a trajectory of the work part in the combined movement of the excavator 100, the user can adjust a distribution of the basic movements constituting the combined movement. The specifications setting part 302E may set specifications for distribution of features on a per plurality of basic movements basis. At this time, for example, as illustrated in FIG. 10, in a case where there are a plurality of features that are reflected (input) in different layers of the trained model LM3 from each other, specifications for the distribution of features on a per plurality of basic movements basis may be set on a per plurality of features basis.

The specifications setting part 302E may adjust the execution order of the plurality of basic movements in the combined movement on a per plurality of basic movements basis. Thus, the user can adjust the execution order of the basic movements constituting the combined movement when generating a trajectory of the work part in the combined movement of the excavator 100. The specifications setting part 302E may adjust conditions such as passing points of the work part in the plurality of basic movements basis constituting the combined movement, on a per plurality of basic movements basis. Thus, when generating the trajectory of the work part in the combined movement of the excavator 100, the user can limit positions through which the work part should pass on a per plurality of basic movements basis while monitoring the image representing a topographical shape around the excavator 100 displayed on the display device 50A.

The trajectory generating part 302F generates a trajectory of the work part in the combined movement of the excavator 100 using the trained models LM2 and LM3, based on the trajectory generated by the trajectory generating part 302D on a per plurality of basic movements basis and the setting results by the specifications setting part 302E.

Specifically, the trajectory generating part 302F extracts features on a per plurality of basic movements basis by using the trained model LM2 based on the data generated on a per plurality of basic movements basis by the trajectory generating part 302D. The trajectory generating part 302F generates a trajectory of the work part in the combined trajectory of the excavator 100 using the trained model LM3, based on the features on a per plurality of basic movements basis and the setting results by the specifications setting part 302E.

For example, as illustrated in FIG. 10, the trajectory generating part 302F adjusts the feature FV based on the combination ratio set on a per plurality of basic movements basis by the specifications setting part 302E, and generates the trajectory of the work part in the combined trajectory of the excavator 100 using the trained model LM3. The trajectory generating part 302F may reflect the setting results by the specifications setting part 302E in data representing the trajectory of the work part in the combined trajectory of the excavator 100 by adjusting the adjustment parameters.

Returning to FIG. 6, the movement control part 302G operates the excavator 100 such that the work part of the bucket 6 moves along the trajectory corresponding to the data generated by the trajectory generating part 302F, in response to the input from the user (operator) received through the input device 52. Specifically, the movement control part 302G can operate the excavator 100 such that the work part of the bucket 6 moves along a target trajectory by controlling the hydraulic control valve 31 while identifying the position of the work part of the bucket 6 from the outputs of the sensors S1 to S5 and the like.

For example, the movement control part 302G operates the excavator 100 such that the work part of the bucket 6 moves along the trajectory corresponding to the trajectory generated by the trajectory generating part 302F in response to the input of the instruction to execute the operation from the user through the input device 52. In addition, in a case where the excavator 100 is remotely operated, the movement control part 302G may operate the excavator 100, in response to an input of an instruction to execute an operation from a user who uses the remote operation assistance device 300, which is received through the communication device 60.

The movement control part 302G may operate the excavator 100 such that the work part of the bucket 6 moves along the trajectory corresponding to the data generated by the trajectory generating part 302F in a manner of assisting the operation of the operator, in response to the operation of the operation device 26 or the remote operation signal.

The display processing part 302H causes the display device 50A to display a screen related to generation of the target trajectory of the work part of the excavator 100.

The screen related to the generation of the trajectory of the work part of the excavator 100 includes, for example, a screen for displaying an image representing the shape of a construction target around the excavator 100, based on the data acquired by the work target shape acquiring part 302B. Thus, the user can check the shape of the construction target around the excavator 100 and determine a movement that needs to be performed by the excavator 100 according to the work content. The image representing the shape of the construction target around the excavator 100 may be, for example, a captured image of the imaging device or a processed image thereof representing the shape of the construction target around the excavator 100, or may be an image of three dimensional data representing the shape of the construction target around the excavator 100. The processed image is, for example, an image obtained by performing a viewpoint conversion process or the like on the captured image acquired by the imaging device 40. For example, as illustrated in FIG. 12, an image TG representing the shape of the construction target around the excavator 100 viewed from the viewpoint of the cabin 10 of the excavator 100 is displayed on the screen. The image TG corresponds to an image of three dimensional data of the construction target around the excavator 100. In this case, the image CG representing the excavator 100 (attachment AT) may be displayed on the screen in a manner that the positional relationship between the image CG and the image TG representing the shape of the construction target around the excavator 100 is adjusted. Further, an image representing the shape of the construction target around the excavator 100 viewed from a predetermined viewpoint around the excavator 100 may be displayed on the screen. Thus, the user can check the shape of the construction target around the excavator 100 from a viewpoint different from a normal viewpoint of the cabin 10. The viewpoint of the image representing the shape of the construction target around the excavator 100 displayed on the screen may be optionally changeable in response to a predetermined input from the user through the input device 52. Thus, the user can check the image representing the shape of the work target around the excavator 100 from a viewpoint that the user desires.

The screen related to the generation of the trajectory of the work part of the excavator 100 includes, for example, an operation screen on which the user performs an operation input for selecting a target work to be performed by the excavator 100 from among a plurality of works through the work selecting part 302C. On the operation screen, the user can perform an operation of selecting a target work to be performed by the excavator 100 from among a plurality of works using the input device 52.

The screen related to the generation of the trajectory of the work part of the excavator 100 includes, for example, an operation screen (setting screen) on which the user performs an operation input for setting the specifications related to a combination of the plurality of basic movements in the combined movement of the excavator 100 through the specifications setting part 302E. On the operation screen, the user can perform an operation of setting specifications related to a combination of a plurality of basic movements in the combined movement of the excavator 100 using the input device 52.

The screen related to the generation of the trajectory of the work part of the excavator 100 includes, for example, an operation screen on which the user performs an operation input for causing the trajectory generating part 302F to generate the trajectory of the work part of the excavator 100 according to the specifications set by the specifications setting part 302E. On the operation screen, the user can perform an operation for instructing the trajectory generating part 302F to generate the trajectory of the work part of the excavator 100 according to the set specifications using the input device 52.

The screen related to the generation of the trajectory of the work part of the excavator 100 includes a screen for displaying the trajectory of the work part of the excavator 100 corresponding to the trajectory generated by the trajectory generating part 302F. At this time, the trajectory of the work part of the excavator 100 may be displayed to be superimposed on the image representing the shape of the construction target around the excavator 100. Further, a moving image of a simulation of the excavator 100 which operates such that the work part moves along the trajectory may be displayed on the screen. Thus, the user can identify in advance how the combined movement of the excavator 100 is performed based on the trajectory of the work part in the combined movement of the excavator 100 corresponding to the trajectory generated by the trajectory generating part 302F. At this time, at the end of the moving image of the simulation of the excavator 100, an image representing the shape of the work target predicted after the work part moves along the trajectory may be displayed on the screen. Thus, the user can identify in advance how the shape of the work target changes by causing the excavator 100 to perform the combined movement, based on the trajectory of the work part in the combined movement of the excavator 100 corresponding to the trajectory generated by the trajectory generating part 302F.

The screen related to the generation of the trajectory of the work part of the excavator 100 includes an operation screen for automatically operating the excavator 100 such that the work part of the excavator 100 moves along the trajectory corresponding to the trajectory generated by the trajectory generating part 302F. On the operation screen, the user can perform an operation of an instruction to operate the excavator 100 such that the work part moves along the trajectory generated by the trajectory generating part 302F, using the input device 52. The movement control part 302G operates the excavator 100 such that the work part of the bucket 6 moves along the trajectory generated by the trajectory generating part 302F, in response to an instruction to execute an operation from the user, which is output in response to an operation on the operation screen. For example, the operation screen is common to a screen that displays the trajectory of the work part of the excavator 100 corresponding to the trajectory generated by the trajectory generating part 302F, and includes an icon corresponding to an instruction to execute an operation. Thus, the user can input an instruction to execute the operation of the excavator 100 through the input device 52 or the like after checking the trajectory of the work part of the excavator 100.

In addition, in a case where the excavator 100 is remotely operated, the display processing part 302H may transmit a screen related to generation of a trajectory of a work part of the excavator 100 to the remote operation assistance device 300 through the communication device 60. Thus, the display processing part 302H can cause the remote operation assistance device 300 (display device) to display a screen related to generation of the target trajectory of the work part of the excavator 100.

In a case where the excavator 100 is remotely operated, functions of the display processing part 302H may be provided in the remote operation assistance device 300. Thus, the display device of the remote operation assistance device 300 can display a screen related to the generation of the trajectory of the work part of the excavator 100. Therefore, the user (operator) of the remote operation assistance device 300 can cause the trajectory generating part 302F to generate the trajectory of the work part in the combined trajectory of the excavator 100 using the screen displayed on the display device of the remote operation assistance device 300. In addition, in a case where the excavator 100 is remotely operated, some or all of the functions of the work target shape acquiring part 302B, the work selecting part 302C, the trajectory generating part 302D, the specifications setting part 302E, the trajectory generating part 302F, and the movement control part 302G may be provided in the remote operation assistance device 300. Some or all of the functions of the work target shape acquiring part 302B, the work selecting part 302C, the trajectory generating part 302D, the specifications setting part 302E, the trajectory generating part 302F, and the movement control part 302G may be transferred to the information processing device 200. Thus, it is possible to reduce the processing load of the excavator 100 and the remote operation assistance device 300 related to the processing related to the generation of the trajectory of the work part in the combined movement of the excavator 100 and the control of the operation of the excavator 100.

[Setting Screen of Specifications for Combination of Plurality of Basic Movements]

Next, a setting screen of specifications related to a combination of a plurality of basic movements in the combined movement of the excavator 100 will be described with reference to FIGS. 13 and 14.

First Example

FIG. 13 is a diagram illustrating a first example (setting screen 1300) of a setting screen of specifications related to a combination of a plurality of basic movements in the combined movement of the excavator 100.

When the excavator 100 is remotely operated, the setting screen 1300 of FIG. 13 is displayed on the display device of the remote operation assistance device 300 and is configured to be operable through the input device of the remote operation assistance device 300. The same applies to a setting screen 1400 of FIG. 14 described below.

As illustrated in FIG. 13, the setting screen 1300 includes images 1301, operation images 1302, and an operation image 1303.

The images 1301 are images schematically representing the basic movements of the excavator 100. The images 1301 include images 1301A to 1301C.

The image 1301A is an image schematically representing the horizontal-pulling movement of the excavator 100.

The image 1301B is an image schematically representing the rolling-compaction movement of the excavator 100.

The image 1301C is an image schematically representing the sweeping-out movement of the excavator 100.

The images 1301A to 1301C are arranged vertically at a position on the left side of the center in the horizontal direction of the setting screen 1300.

The operation images 1302 and 1303 are images that can be operated through the input device 52.

The operation images 1302 are operation images for adjusting the distribution of the combination of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement. The operation images 1302 include operation images 1302A to 1302C.

The operation image 1302A is a slide bar that can adjust a composition ratio (combination ratio) of the horizontal-pulling movement in the combined movement of the excavator 100 between 0% and 100%. The operation image 1302A is arranged on the right side of the image 1301A.

The operation image 1302B is a slide bar that can adjust the composition ratio (combination ratio) of the rolling-compaction movement in the combined movement of the excavator 100 between 0% and 100%. The operation image 1302B is arranged on the right side of the image 1301B.

The operation image 1302C is a slide bar that can adjust the composition ratio (combination ratio) of the sweeping-out movement in the combined movement of the excavator 100 between 0% and 100%. The operation image 1302C is arranged on the right side of the image 1301C.

The operation images 1302A to 1302C are arranged vertically at a position on the right side of the center in the horizontal direction of the setting screen 1300.

The combination ratio of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement is adjusted such that the total of the combination ratio of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement is 100%. For example, all the slide bars of the operation images 1302A to 1302C are set to 0% in the initial state, and when any of the slide bars is designated to a percentage larger than 0%, the other slide bars are restricted to be movable only in a range not exceeding 100% in total.

The operation image 1303 is an operation image for determining the combination ratio of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement in the combined movement of the excavator 100, which are designated by the operation images 1302, and for causing the trajectory generating part 302F to execute the generation process of the trajectory of the work part.

In this example, the slide bar of the operation image 1302B is set to 0%, and the operation images 1302A and 1302B are both designated to about 50%. In this state, by operating the operation image 1303, the user can create data representing the trajectory of the work part in the combined movement by the combination of the sweeping-out movement and the horizontal-pulling movement of the excavator 100 along the combination ratio designated by the operation images 1302 (see FIG. 11).

When one of the slide bars of the operation images 1302A to 1302C is designated as 100% and the other two slide bars are designated as 0%, the trajectory generating part 302F can generate the trajectory of the work part by the basic movement, not the combined movement of the excavator 100.

Second Example

FIG. 14 is a diagram illustrating a first example (setting screen 1400) of a setting screen of specifications related to a combination of a plurality of basic movements in the combined movement of the excavator 100.

As illustrated in FIG. 13, the setting screen 1400 includes images 1401, operation images 1402, and an operation image 1403.

The images 1401 is an image schematically representing the basic movement of the excavator 100. The images 1401 include image 1401A to 1401C.

The image 1401A is an image schematically representing the horizontal-pulling movement of the excavator 100. The image 1401A is arranged at a position in the center in the horizontal direction of the setting screen 1400 and above the center in the vertical direction.

The image 1401B is an image schematically representing the rolling-compaction movement of the excavator 100. The image 1401B is arranged on the right side of the center in the horizontal direction of the setting screen 1400 and on the lower side of the center in the vertical direction.

The image 1401C is an image schematically representing the sweeping-out movement of the excavator 100. The image 1401C is arranged on the left side of the center in the horizontal direction of the setting screen 1400 and on the lower side of the center in the vertical direction. The image 1401A to 1401C are arranged to form an approximately equilateral triangle on the setting screen 1400.

The operation images 1402 and 1403 are images that can be operated through the input device 52.

The operation images 1402 are operation images for adjusting the distribution of the combination of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement. The operation images 1402 include operation images 1402A to 1402D.

The operation image 1402A is a circular image representing a state where the composition ratio (combination ratio) of the horizontal-pulling movement of the excavator 100 in the combined movement of the excavator 100 is 100%, that is, a state where the excavator 100 performs the horizontal-pulling movement instead of the combined movement. The operation image 1402A is arranged adjacent to the lower side of the image 1401A.

The operation image 1402B is a circular image representing a state where the composition ratio for the rolling-compaction movement in the combined movement of the excavator 100 is 100%, that is, a state where the excavator 100 performs the rolling-compaction movement instead of the combined movement. The operation image 1402B is arranged on the left side of the image 1401A.

The operation image 1402C is a circular image representing a state where the composition ratio for the sweeping-out movement in the combined movement of the excavator 100 is 100%, that is, a state where the excavator 100 performs the sweeping-out movement instead of the combined movement. The operation image 1402C is arranged on the right side of the image 1401C.

The operation images 1402A to 1402C are arranged to form an approximately equilateral triangle, and the operation images 1402A and 1402B, the operation images 1402B and 1402C, and the operation images 1402C and 1402A are connected by straight lines. Hereinafter, the operation images 1402A to 1402C may be referred to as “operation image 1402X” without being individually distinguished.

The operation image 1402D is an image of a circular cursor for designating a composition ratio (combination ratio) of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement in the combined movement of the excavator 100. The operation image 1402D can be moved on the triangle formed by the operation images 1402A to 1402C or inside a range of the triangle.

The composition ratio (combination ratio) of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement in the combined movement of the excavator 100 is uniquely designated by a position of a cursor of the operation image 1402D on or inside the triangle formed by the operation images 1402A to 1402C. Specifically, the combination ratio is designated by the position of the cursor of the operation image 1402D such that the smaller the distance between one operation image 1402X and the operation image 1402D relative to the distance between each of the other two operation images 1402X and the operation image 1402D, the greater the combination ratio for a basic movement corresponding to one operation image. Conversely, the combination ratio is designated by the position of the cursor of the operation image 1402D such that the greater the distance between one operation image 1402X and the operation image 1402D relative to the distance between each of the other two operation images 1402X and the operation image 1402D, the smaller the combination ratio for a basic movement corresponding to one operation image.

The operation image 1403 is an operation image for determining the combination ratio of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement in the combined movement of the excavator 100, which are designated by the operation images 1402, and for causing the trajectory generating part 302F to execute the generation process of the trajectory of the work part.

For example, as illustrated in FIG. 14, when the operation image 1402D is on one side of the triangle between the operation images 1402A and 1402C, the combination ratio for the rolling-compaction movement is designated to 0%. Then, the combination ratio of the horizontal-pulling movement and the sweeping-out movement in the combined movement of the excavator 100 are designated by the position of the operation image 1402A on the line segment (one side) between the operation images 1402D and 1402C. In this state, when the operation image 1403 is operated, the trajectory generating part 302F generates a trajectory of the work part in the combined movement of the combination of the horizontal-pulling movement and the sweeping-out movement of the excavator 100 along the combination ratio corresponding to the position of the operation image 1402D. Thus, the user can create data representing the trajectory of the work part by the combined movement of the horizontal-pulling movement and the sweeping-out movement, from among the horizontal-pulling movement, the sweeping-out movement, and the rolling-compaction movement of the excavator 100 (see FIG. 11).

In addition, in a case where the operation image 1402D is on one side of the triangle between the operation images 1402A and 1402B, the combination ratio for the sweeping-out movement is designated as 0%. Then, the combination ratio of the horizontal-pulling movement and the rolling-compaction movement in the combined movement of the excavator 100 is designated by the position of the operation image 1402A on the line segment (one side) between the operation images 1402D and 1402B. In this state, when the operation image 1403 is operated, the trajectory generating part 302F generates a trajectory of the work part in the combined movement of the combination of the horizontal-pulling movement and the rolling-compaction movement of the excavator 100 along the combination ratio corresponding to the position of the operation image 1402D. Thus, the user can create data representing the trajectory of the work part by a combined movement of the horizontal-pulling movement and the rolling-compaction movement among the horizontal-pulling movement, the sweeping-out movement, and the rolling-compaction movement of the excavator 100.

Further, when the operation image 1402D is on one side of the triangle between the operation images 1402B and 1402C, the combination ratio for the horizontal-pulling movement is specified as 0%. Then, the combination ratio of the rolling-compaction movement and the sweeping-out movement in the combined movement of the excavator 100 is designated by the position of the operation image 1402B on the line segment (one side) between the operation images 1402D and 1402C. In this state, when the operation image 1403 is operated, the trajectory generating part 302F generates a trajectory of the work part in a combined movement of a combination of the rolling-compaction movement and the sweeping-out movement of the excavator 100 along the combination ratio corresponding to the position of the operation image 1402D. Thus, the user can create data representing the trajectory of the work part by the combined movement of the rolling-compaction movement and the sweeping-out movement among the horizontal-pulling movement, the sweeping-out movement, and the rolling-compaction movement of the excavator 100.

In addition, in a case where the operation image 1402D is inside the triangle formed by the operation images 1402A to 1402C, the combination ratio for each of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement is designated to be greater than 0%. Specifically, the combination ratio of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement is designated to be 100% in total, according to the relative magnitude relationships in distance between the operation image 1402D and each of the operation images 1402A to 1402C. In this state, when the operation image 1403 is operated, the trajectory generating part 302F generates a trajectory of the work part in a combined movement of a combination of the horizontal-pulling movement, the rolling-compaction movement, and the sweeping-out movement of the excavator 100 along a combination ratio corresponding to the position of the operation image 1402D. Thus, the user can create data representing the trajectory of the work part by a combined movement from among all the combinations of the horizontal-pulling movement, the sweeping-out movement, and the rolling-compaction movement of the excavator 100.

When the operation image 1402D is designated at the same position as any one of the operation images 1402A to 1402C, the trajectory generating part 302F can generate a trajectory of the work part by the corresponding basic movement, not by the combined movement of the excavator 100.

[Process for Generating Trajectory of Work Part of Excavator]

Next, a process related to generation of a trajectory of a work part of the excavator 100 will be described with reference to FIG. 15.

FIG. 15 is a flowchart schematically illustrating an example of a process related to generation of a trajectory of a work part of the excavator 100.

The flowchart of FIG. 15 is started when the controller 30 receives a predetermined input for performing the process related to generation of a trajectory of a work part of the excavator 100 from the input device 52 or the remote operation assistance device 300.

As illustrated in FIG. 15, in step S102, the work target shape acquiring part 302B acquires the latest captured image of the imaging device 40, and acquires the shape of the work target around the excavator 100 based on the captured image.

When the process of step S102 is completed, the controller 30 proceeds to step S104.

In step S104, the display processing part 302H causes the display device 50A or the display device of the remote operation assistance device 300 to display an image representing the shape of the work target around the excavator 100 (see FIG. 12), based on the data acquired in step S102.

When the process of step S104 is completed, the controller 30 proceeds to step S106.

In step S106, the display processing part 302H causes the display device 50A or the display device of the remote operation assistance device 300 to display a screen for selecting one work from among a plurality of work candidates. For example, the display processing part 302H displays options of the plurality of work candidates in a superimposed manner on the image representing the shape of the work target around the excavator 100.

When the process of step S106 is completed, the controller 30 proceeds to step S108.

In step S108, the work selecting part 302C selects one work from among the plurality of work candidates (for example, the land leveling work, the digging work, the slope construction work, and the like) in response to a predetermined input on the selection screen in step S106, which is received through the input device 52 or the like.

When the process of step S108 is completed, the controller 30 proceeds to step S110.

In step S110, the trajectory generating part 302D generates, based on the data acquired in step S102, trajectories of the work part on a per plurality of basic movements basis corresponding to the work selected in step S106, using the trained model LM1.

When the process of step S110 is completed, the controller 30 proceeds to step S112.

In step S112, the display processing part 302H causes the display device 50A or the display device of the remote operation assistance device 300 to display a setting screen of specifications related to a combination of a plurality of basic movements.

When the process of step S112 is completed, the controller 30 proceeds to step S114.

In step S114, the specifications setting part 302E sets the specifications related to the combination of the plurality of basic movements in response to the predetermined input on the setting screen in step S112, which is received through the input device 52 or the like.

When the process of step S114 is completed, the controller 30 proceeds to step S116.

In step S116, the trajectory generating part 302F generates a trajectory of the work part by the combined movement of the excavator 100 using the trained models LM2 and LM3 according to the trajectories generated in step S110 and the setting contents in step S114.

When the process of step S116 is completed, the controller 30 proceeds to step S118.

In step S118, the display processing part 302H causes the display device 50A or a display part of the remote operation assistance device 300 to display an image representing the trajectory of the work part by the combined movement of the excavator 100 corresponding to the data generated in step S116.

When the process of step S118 is completed, the controller 30 proceeds to step S120.

In step S120, the controller 30 determines whether or not an input instructing execution of a movement of the excavator 100 for moving the work part of the bucket 6 along the trajectory corresponding to the data generated in step S118 has been received. When the input for instructing the execution of the movement of the excavator 100 is received, the controller 30 proceeds to step S122. On the other hand, when another input, in particular, an input for generating the trajectory of the work part by the combined movement of the excavator 100 is received again, the controller 30 returns to step S106.

In step S122, the movement control part 302G controls the hydraulic control valve 31 to automatically perform a movement of the excavator 100 such that the work part of the bucket 6 moves along the trajectory corresponding to the data generated in the process of the latest step S116.

When the process of step S122 is completed, the controller 30 ends the process of the flowchart of this time.

At the time of completion of the process of step S122, the excavator 100 (attachment AT) may be in a state where the work part of the bucket 6 is at the end point of the trajectory. The attitude state may be returned to the attitude state before the start of the process of step S122.

In this way, in the present example, the assistance device 150 (the controller 30) can generate the trajectory of the work part by the combined movement of the excavator 100 according to the specifications related to the combination of the plurality of basic movements set by the user. Therefore, the work efficiency of the excavator 100 can be improved.

Functional Effects

Next, the functional effects of the assistance device, the work machine, the assistance system, and the program according to the present embodiment will be described.

In the present embodiment, the assistance device includes an input part and a display part. The assistance device is, for example, the assistance device 150, the information processing device 200, or the remote operation assistance device 300. The input part is, for example, the input device 52 or the input device of the remote operation assistance device 300. Specifically, the input part receives an input from a user. The display part determines, in response to the input from the input part, specifications related to a combination of a plurality of movements of different types from each other of the work machine, and displays a first operation screen for generating a trajectory of the work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements. The work machine is, for example, the excavator 100 described above. The work machine may be the crane, the forklift, or the road machine described above. The display part is, for example, the display device 50A or the display device of the remote operation assistance device 300. The first operation screen is, for example, the setting screen 1300 or the setting screen 1400 described above.

In the present embodiment, a program causes an information processing device including an input part and a display part to execute a display step. Specifically, in the display step, specifications related to a combination of a plurality of movements of different types from each other of the work machine are determined in response to the input from the input part, and a first operation screen for generating a trajectory of the work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements is displayed on the display part. The display step is, for example, a step S112.

More specifically, on the first operation screen, the user may be able to perform an operation of determining the specifications through the input part.

Thus, the user can generate the trajectory of the work part by the combined movement of the excavator 100, which is a combination of a plurality of movements, by operating the first operation screen using the input part. Therefore, the user can proceed with the work of the work machine by operating the work machine such that the work part moves along the trajectory, or by activating the automatic driving function of the work machine such that the work part moves along the trajectory. Therefore, it is possible to improve the work efficiency when the work is performed by combining the plurality of basic movements of the work machine.

In the present embodiment, each of the plurality of movements may be a movement performed by the work machine using the work attachment in a predetermined work. The work attachment is, for example, the attachment AT described above.

Thus, the user can generate the trajectory of the work part by the combined movement of the work machine according to the predetermined work performed by the work machine.

In the present embodiment, the predetermined work may be land leveling work, digging work, or slope construction work. When the predetermined work is the land leveling work, the plurality of movements may include at least two of the horizontal-pulling movement, the rolling-compaction movement, the broom-turning movement, the digging movement, and the earth removal movement. Further, when the predetermined work is the digging work, the plurality of movements may include at least two of the digging movement, the boom raising-slewing movement, the boom lowering-slewing movement, the earth removal movement, and the broom-turning movement. Further, the plurality of movements may include an earth cutting movement and a rolling-compaction movement when the predetermined work is slope construction work.

Thus, the user can generate the trajectory of the work part by the combined movement of the work machine according to the land leveling work, the digging work, or the slope construction work.

In the present embodiment, the trajectory of the work part may be a trajectory of a predetermined part set in an end attachment at the distal end of the work attachment of the work machine. The end attachment is, for example, the bucket 6.

Thus, the user can generate the trajectory of the predetermined part of the end attachment at the distal end of the work attachment as a work part that abuts on the work target in the work machine.

In the present embodiment, the above-described specifications may include distribution of a combination of a plurality of movements in a combined movement of the work machine.

Thus, the user can cause the assistance device to generate a more appropriate trajectory of the work part by the combined movement of the excavator 100 by appropriately adjusting the distribution (for example, the composition ratio) of the combination of the plurality of movements in the combined movement of the work machine.

In the present embodiment, the display part may display an image representing the shape of the work target around the work machine, and may display the trajectory generated through the first operation screen so as to be superimposed on the image representing the shape of the work target around the work machine.

More specifically, the image representing the shape of the work target may be a captured image around the work machine or a processed image of the captured image, or an image of three dimensional data of the work target around the work machine.

Thus, the user can check the validity of the generated trajectory while comparing the image representing the shape of the work target around the work machine with the generated trajectory.

In the present embodiment, the display part may display a moving image of a simulation of the work machine in which the work part moves along the trajectory generated through the first operation screen, superimposed on the image representing the shape of the work target.

Thus, the user can check in advance the state in which the excavator 100 performs the combined movement such that the work part moves along the generated trajectory.

In the present embodiment, the display part may display an image representing a shape of the work target predicted after the work part moves along the trajectory generated through the first operation screen.

Thus, the user can check in advance how the shape of the work target changes after the excavator 100 performs the combined movement such that the work part moves along the generated trajectory.

In the present embodiment, the display part may display, in response to the input from the input part, a second operation screen for operating the work machine so as to move the work part along the trajectory generated through the first operation screen.

More specifically, on the second operation screen, the user may be able to perform an operation for instructing the work machine to operate such that the work part moves along the trajectory generated through the first operation screen, using the input part.

Thus, the user can cause the excavator 100 to automatically execute the combined movement such that the work part moves along the generated trajectory. Therefore, for example, even when the user (operator) is not a skilled operator, the work part of the work machine can be moved along the generated trajectory, and as a result, the work efficiency of the work machine can be further improved.

In the present embodiment, the work machine may include the above-described assistance device. That is, the work machine may include an input part that receives an input from a user, and a display part that determines specifications related to a combination of a plurality of movements of the work machine different from each other according to the input from the input part and displays a first operation screen for generating a trajectory of the work part of the work machine by a combined movement, the combined movement being obtained by combining the plurality of movements.

Thus, the user (operator) getting into the work machine can generate the trajectory of the work part by the combined movement of the excavator 100 using the assistance device.

In the present embodiment, the assistance system may include a work machine and the above-described assistance device. That is, the assistance system is, for example, the activation assistance system SYS described above.

Thus, a user (operator) outside the work machine can generate a trajectory of the work part by the combined movement of the excavator 100 using the assistance device.

Further Disclosure

The following is further disclosed in relation to the above-described embodiment.

(1)

A work assistance system including:

- an acquiring part configured to acquire data related to a shape of a work target around a work machine; and
- a first generating part configured to generate, based on the data acquired by the acquiring part, a trajectory of a work part of the work machine by a combined movement, the combined movement being obtained by combining a plurality of movements of the work machine.

The work assistance system is, for example, the above-described activation assistance system SYS. The work machine is, for example, the excavator 100 described above. The acquiring part is, for example, the work target shape acquiring part 302B described above. The first generating part is the trajectory generating part 302F described above.

(2)

The work assistance system according to (1), further including:

- a second generating part configured to generate trajectories of the work part on a per plurality of movements basis, based on the data acquired by the acquiring part,
- wherein the first generating part generates the trajectory of the work part by the combined movement, based on the trajectories of the work part on a per plurality of movements basis generated by the second generating part.

The second generating part is, for example, the trajectory generating part 302D described above.

(3)

The work assistance system according to (1) or (2), further including:

- a selecting part configured to select the plurality of movements from among movements of the work machine, based on the data acquired by the acquiring part.

The selecting part is, for example, the work selecting part 302C described above.

(4)

The work assistance system according to (2), further including:

- an input part configured to receive an input from a user,
- wherein the first generating part generates the trajectory of the work part by the combined movement, in response to an input related to specifications of a combination of the plurality of movements from the input part.

The input part is, for example, the input device 52 or an input device of the remote operation assistance device 300.

(5)

The work assistance system according to (4), further including:

- an extraction part configured to extract features on a per plurality of movements basis generated by the second generating part,
- wherein the first generating part adjusts a combination ratio for the features on a per plurality of movements basis, in response to the input related to the specifications of the combination of the plurality of movements from the input part, and generates the trajectory of the work part by the combined movement, based on the adjusted features.

The extraction part is, for example, the trajectory generating part 302F described above.

(6)

The work assistance system according to any one of (1) to (5), further including:

- a display part configured to display an image representing the shape of the work target based on the data acquired by the acquiring part
- wherein the display part displays the trajectory generated by the first generating part so as to be superimposed on the image representing the shape of the work target.

The display part is, for example, the display device 50A or the display device of the remote operation assistance device 300.

(7)

The work assistance system according to (6),

- wherein the display part displays a moving image in which the work part moves along the trajectory generated by the first generating part, the moving image being superimposed on the image representing the shape of the work target.
  
  (8)

The work assistance system according to (6) or (7),

- wherein the display part displays an image representing the shape of the work target predicted after the work part moves along the trajectory generated by the first generating part.
  
  (9)

The assistance system according to any one of (1) to (8), further including:

- an input part configured to receive an input from a user; and
- a control part configured to automatically move the work machine, based on the trajectory generated by the first generating part, in response to a predetermined input from the input part.

The control part is, for example, the movement control part 302G described above.

(10)

An information processing device including:

- a first generating part configured to generate a trajectory of a work part of a work machine by a combined movement, based on data acquired by an acquiring part, the combined movement being obtained by combining a plurality of movements of the work machine, the acquiring part being configured to acquire data related to a shape of a work target around the work machine.

The information processing device is, for example, the controller 30, the information processing device 200, or the remote operation assistance device 300 described above.

(11)

A work machine including:

- an acquiring part configured to acquire data related to a shape of a work target around a work machine;
- a first generating part configured to generate, based on the data acquired by the acquiring part, a trajectory of a work part of the work machine by a combined movement, the combined movement being obtained by combining a plurality of movements of the work machine.
  
  (12)

A program for causing an information processing device to execute a first generating step of generating, based on data acquired by an acquiring part, a trajectory of a work part of a work machine by a combined movement, the combined movement being obtained by combining a plurality of movements of the work machine, the acquiring part being configured to acquire data related to a shape of a work target around the work machine.

Although the embodiments have been described in detail, the present disclosure is not limited to such specific embodiments, and various modifications and changes can be made within the scope of the gist described in the claims.

Finally, the present application claims priority to Japanese Patent Application No. 2022-127437 filed on Aug. 9, 2022, the entire contents of which are incorporated herein by reference.

	Number	Date	Country
Parent	PCT/JP2023/029009	Aug 2023	WO
Child	19046666		US

ASSISTANCE DEVICE, WORK MACHINE, AND ASSISTANCE SYSTEM

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (1)

CROSS-REFERENCE TO RELATED APPLICATIONS

Continuations (1)