Patent Application
20230036534
The present disclosure relates to a remote operation system for a work machine.
In a technical field related to work machines, a technique for remotely operating a work machine is known. In the remote operation of the work machine, an image of a work site where the work machine operates is captured by an imaging device. The image captured by the imaging device is transmitted to a remote place and displayed on a display device disposed at the remote place. The operator at the remote place remotely operates the work machine while viewing the image displayed on the display device. Patent Literature 1 discloses a technique in which an imaging device is disposed in a driver's cab on a swing body and captures an image in front of the driver's cab.
There is a possibility that an abnormality occurs in an image transmission path from a work machine to a remote place. If the operator at the remote place continues the remote operation without recognizing the abnormality of the image transmission path even though the abnormality has occurred in the image transmission path, productivity at the work site may be reduced.
An object of the present disclosure is to allow an operator at a remote place to recognize the presence or absence of an abnormality in an image transmission path.
according to an aspect of the present invention, a remote operation system for a work machine, comprises, at a remote place of the work machine: an image data reception unit that receives a first image in a first imaging range and a second image in a second imaging range at least partially overlapping the first imaging range; and a display control unit that causes a display device to display the first image and the second image including working equipment of the work machine in an overlapping range between the first imaging range and the second imaging range.
According to the present disclosure, it is possible to allow an operator at a remote place to recognize the presence or absence of an abnormality in an image transmission path.
Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of the embodiments described below can be appropriately combined. In addition, some components may not be used.
In the embodiments, the positional relationship of each part will be described using terms of “left”, “right”, “front”, “rear”, “upper”, and “lower”. These terms indicate relative positions or directions with respect to the center of a swing body 3 of a work machine 1.
[Remote Operation System]
At least a part of the remote operation system 100 is disposed in a remote operation room 200. The remote operation room 200 is built at a remote place away from the work site. The remote operation system 100 includes a remote operation device 40, a display device 50, and a control device 60.
The remote operation device 40 is disposed in the remote operation room 200. The remote operation device 40 is operated by an operator in the remote operation room 200. The operator can operate the remote operation device 40 while seated on an operation seat 45.
The display device 50 is disposed in the remote operation room 200. The display device 50 displays an image of the work site. The operator in the remote operation room 200 cannot directly visually recognize the situation of the work site. The operator in the remote operation room 200 can visually recognize the situation of the work site via the display device 50.
The operator operates the remote operation device 40 while viewing the image of the work site displayed on the display device 50. The work machine 1 is remotely operated by the remote operation device 40.
The control device 60 is disposed in the remote operation room 200. The control device 60 includes a computer system.
The work machine 1 includes a control device 300. The control device 300 includes a computer system.
The control device 60 and the control device 300 communicate with each other via a communication system 400. Examples of the communication system 400 include the Internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network. The communication system 400 may include a relay station that relays data to be communicated.
[Work Machine]
As illustrated in
The traveling body 2 travels while supporting the swing body 3. The swing body 3 is a vehicle body of the work machine 1. The traveling body 2 is disposed below the swing body 3. The traveling body 2 supports the swing body 3 in a swingable manner. The traveling body 2 includes a drive wheel 2A, a driven wheel 2B, and a crawler belt 2C supported by the drive wheel 2A and the driven wheel 2B. Each of the drive wheel 2A and the driven wheel 2B rotates about the rotation axis DX. A pair of drive wheels 2A, a pair of driven wheels 2B, and a pair of crawler belts 2C are provided. The crawler belt 2C is rotated by the rotation of the drive wheel 2A. When the crawler belt 2C rotates, the traveling body 2 travels.
The swing body 3 is swingable about the swing axis RX while being supported by the traveling body 2. The swing axis RX extends in the vertical direction. The swing body 3 includes a driver's cab 3A, a lower deck 3B, steps 3C, and an upper deck 3D. The driver's cab 3A is an internal space of the swing body 3 on which an operator can board. The driver's cab 3A is disposed at a front portion and an upper portion of the swing body 3. The lower deck 3B is disposed at a rear portion and a lower portion of the swing body 3. The upper deck 3D is disposed at a front portion and an upper portion of the swing body 3. The steps 3C connect the lower deck 3B and the upper deck 3D. The upper deck 3D is disposed so as to surround the driver's cab 3A. At least a part of the upper deck 3D is disposed in front of the driver's cab 3A. A palisade handrail 3E is disposed on each of the lower deck 3B, the steps 3C, and the upper deck 3D.
Each of the lower deck 3B, the steps 3C, and the upper deck 3D includes a passage through which an operator can pass. The operator can board the driver's cab 3A by passing through the lower deck 3B, the steps 3C, and the upper deck 3D.
In addition, the swing body 3 includes a ladder 3F. The ladder 3F is connected to the upper deck 3D.
The working equipment 4 is attached to the front portion of the swing body 3. The working equipment 4 is disposed in front of the swing axis RX. The working equipment 4 is operable to extend forward. The working equipment 4 includes a boom 4A connected to the swing body 3, an arm 4B connected to the boom 4A, and a bucket 4C connected to the arm 4B. The proximal end portion of the boom 4A is connected to the front portion of the swing body 3 via a pin. The proximal end portion of the arm 4B is connected to the distal end portion of the boom 4A via a pin. The proximal end portion of the bucket 4C is connected to the distal end portion of the arm 4B via a pin. The bucket 4C has a tip blade 4D. The bucket 4C excavates the work target.
The boom 4A is connected to the front portion of the swing body 3 so as to be rotatable about the boom rotation axis AX. The arm 4B is connected to the boom 4A so as to be rotatable about the arm rotation axis BX. The bucket 4C is connected to the arm 4B so as to be rotatable about the bucket rotation axis CX.
The boom rotation axis AX, the arm rotation axis BX, and the bucket rotation axis CX are parallel. Each of the boom rotation axis AX, the arm rotation axis BX, and the bucket rotation axis CX extends in the vehicle width direction of the swing body 3.
In the embodiment, the work machine 1 is a loading excavator. The loading excavator refers to an excavator in which the bucket 4C is attached to the arm 4B such that the tip blade 4D of the bucket 4C faces forward.
The hydraulic cylinder 5 includes a boom cylinder 5A that drives the boom 4A, an arm cylinder 5B that drives the arm 4B, and a bucket cylinder 5C that drives the bucket 4C. The proximal end portion of the boom cylinder 5A is connected to the swing body 3. The distal end portion of the boom cylinder 5A is connected to the boom 4A. The proximal end portion of the arm cylinder 5B is connected to the boom 4A. The distal end portion of the arm cylinder 5B is connected to the arm 4B. The proximal end portion of the bucket cylinder 5C is connected to the boom 4A. The distal end portion of the bucket cylinder 5C is connected to the bucket 4C.
[Imaging Device]
The imaging device 30 images the work site and acquires an image of the work site. The imaging device 30 is disposed on the swing body 3.
Examples of the image of the work site acquired by the imaging device 30 include an image of the work target of the work machine 1, an image of at least a part of the work machine 1, an image of a structure existing in the work site, an image of a work machine different from the work machine 1, and an image of an operator working in the work site. In the embodiment, the image of the work target of the work machine 1 includes an image of the excavation target of the working equipment 4.
The imaging device 30 includes an optical system and an image sensor that receives light passing through the optical system. The image sensor includes a couple charged device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.
In the embodiment, the imaging device 30 includes a first imaging device 31 that is disposed on the swing body 3 and images a first imaging range M1, and a second imaging device 32 that is disposed on the swing body 3 and images a second imaging range M2. Each of the first imaging device 31 and the second imaging device 32 is fixed to the swing body 3. The relative position between the first imaging device 31 and the second imaging device 32 is fixed. The second imaging device 32 images the second imaging range M2 in parallel with the imaging of the first imaging range M1 by the first imaging device 31.
In the vertical direction of the swing body 3, the second imaging device 32 is disposed below the first imaging device 31. That is, in the vertical direction of the swing body 3, the incident surface of the optical system of the second imaging device 32 is disposed below the incident surface of the optical system of the first imaging device 31.
In the left-right direction of the swing body 3, the first imaging device 31 is disposed at the same position as the second imaging device 32. The left-right direction of the swing body 3 corresponds to the vehicle width direction of the swing body 3. That is, in the left-right direction of the swing body 3, the incident surface of the optical system of the first imaging device 31 is disposed at the same position as the incident surface of the optical system of the second imaging device 32.
In the front-rear direction of the swing body 3, the first imaging device 31 is disposed behind the second imaging device 32. That is, in the front-rear direction of the swing body 3, the incident surface of the optical system of the first imaging device 31 is disposed behind the incident surface of the optical system of the second imaging device 32.
In the embodiment, the vertical direction is a direction parallel to the swing axis RX. The left-right direction is a direction parallel to the boom rotation axis AX. The front-rear direction is a direction orthogonal to both the boom rotation axis AX and the swing axis RX. A direction in which the swing body 3 exists with reference to the ground contact surface of the traveling body 2 is an upward direction, and a direction opposite to the upward direction is a downward direction. One of the left-right directions with reference to the swing axis RX is the right, and the opposite direction of the right is the left. The direction in which the working equipment 4 exists with reference to the swing axis RX is the front, and the opposite direction of the front is the rear.
The first imaging device 31 images the front of the swing body 3. The first imaging range M1 is set in front of the swing body 3. The second imaging device 32 images the front of the swing body 3. The second imaging range M2 is set in front of the swing body 3. As illustrated in
The first imaging range M1 includes a front space SP1 of the swing body 3. The front space SP1 includes a space in front of the driver's cab 3A, a space obliquely above the front side of the driver's cab 3A, a space obliquely below the front side, a space obliquely left on the front side, and a space obliquely right on the front side.
The second imaging range M2 includes a front lower space SP2 below the front space SP1. The front lower space SP2 is a space below the front side of the driver's cab 3A. The front lower space SP2 includes a space in front of the traveling body 2. In addition, the front lower space SP2 includes a space on the right side of the front space SP1.
The second imaging range M2 is set such that an image suitable for excavation work by the work machine 1 can be obtained. The second imaging range M2 is set to include, for example, an excavation target such as ground or cliff from the lower side (foot) in front of the swing body 3. In addition, the second imaging range M2 is set to include, for example, the bucket 4C for performing excavation work. Note that the second imaging range M2 may be set to include a space in front of the driver's cab 3A and at substantially the same height as the driver's cab 3A in the vertical direction. Note that the second imaging range M2 may be set to include a space in front of the driver's cab 3A and above the driver's cab 3A in the vertical direction.
The first imaging device 31 is disposed inside the driver's cab 3A. In the embodiment, the first imaging device 31 is disposed on the front portion and the upper portion of the driver's cab 3A. The first imaging device 31 images the front space SP1 of the swing body 3 via the windshield of the driver's cab 3A. Note that the first imaging device 31 may be disposed behind the driver's cab 3A or may be disposed below the driver's cab 3A.
The second imaging device 32 is disposed on the upper deck 3D of the swing body 3. At least a part of the upper deck 3D is disposed in front of the driver's cab 3A. The second imaging device 32 is disposed on the lower surface of the upper deck 3D in front of the swing body 3.
The first imaging device 31 is disposed in the driver's cab 3A such that the optical axis OA1 of the optical system of the first imaging device 31 extends in the front-rear direction. The second imaging device 32 is disposed on the lower surface of the upper deck 3D such that the optical axis OA2 of the optical system of the second imaging device 32 is inclined downward toward the front. In addition, as illustrated in
At least a part of the working equipment 4 is included in the first imaging range M1. That is, the first imaging range M1 is set to include at least a part of the working equipment 4. The bucket 4C of the working equipment 4 moves in the vertical direction within a predetermined movable range. The first imaging range M1 is set to include the bucket 4C that moves in the vertical direction.
In addition, at least a part of the work target of the work machine 1 is included in the first imaging range M1. The first imaging range M1 is set to include at least a part of the work target of the work machine 1.
At least a part of the traveling body 2 is movable to the second imaging range M2. That is, the second imaging range M2 is set so as to be able to include at least a part of the traveling body 2. When the swing body 3 swings such that the rotation axis DX of the traveling body 2 and the boom rotation axis AX become parallel to each other, the second imaging range M2 is set to include a front portion of the crawler belt 2C of the traveling body 2. Note that the traveling body 2 may not be included in the second imaging range M2 depending on, for example, the swing angle of the swing body 3.
In addition, the second imaging range M2 is set to include the bucket 4C of the working equipment 4 that has moved downward. As described above, the bucket 4C moves in the vertical direction within the predetermined movable range. The second imaging range M2 is set to include the bucket 4C moved to the lower portion of the movable range.
In addition, the second imaging range M2 is set to include at least a part of the work target of the work machine 1.
In addition, the second imaging range M2 is set to include the ground GR on which the traveling body 2 travels.
In the embodiment, the first imaging range M1 and at least a part of the second imaging range M2 overlap. As illustrated in
In the following description, an image in the first imaging range M1 captured by the first imaging device 31 will be appropriately referred to as a first image P1, and an image in the second imaging range M2 captured by the second imaging device 32 will be appropriately referred to as a second image P2.
[Remote Operation Room]
The remote operation device 40 is operated by an operator seated on the operation seat 45. The operator sits on the operation seat 45 so as to face the display screen of the display device 50. The operator operates the remote operation device 40 while viewing the display screen of the display device 50.
An operation signal generated by operating the remote operation device 40 is transmitted to the control device 300 of the work machine 1 via the control device 60 and the communication system 400. The control device 300 operates the work machine 1 based on the operation signal acquired via the communication system 400. The operation of the work machine 1 includes at least one of the operation of the traveling body 2, the operation of the swing body 3, and the operation of the working equipment 4.
The operation of the traveling body 2 includes a forward operation and a backward operation of the traveling body 2. The operation of the swing body 3 includes a left swing operation and a right swing operation of the swing body 3. The operation of the working equipment 4 includes a raising operation of the boom 4A, a lowering operation of the boom 4A, a dumping operation of the arm 4B, an excavation operation of the arm 4B, an excavation operation of the bucket 4C, and a dumping operation of the bucket 4C.
The remote operation device 40 includes a left working lever 41 and a right working lever 42 operated for the operation of the swing body 3 and the working equipment 4, and a left traveling pedal 43 and a right traveling pedal 44 operated for the operation of the traveling body 2. The left working lever 41 is operated by the operator's left hand. The right working lever 42 is operated by the operator's right hand. The left traveling pedal 43 is operated by the operator's left foot. The right traveling pedal 44 is operated by the operator's right foot.
The left working lever 41 is disposed on the left side of the operation seat 45. The right working lever 42 is disposed on the right side of the operation seat 45. As an example, when the left working lever 41 is operated in the front-rear direction, the arm 4B performs a dumping operation or an excavation operation. When the left working lever 41 is operated in the left-right direction, the swing body 3 performs a left swing operation or a right swing operation. When the right working lever 42 is operated in the left-right direction, the bucket 4C performs an excavation operation or a dumping operation. When the right working lever 42 is operated in the front-rear direction, the boom 4A performs a lowering operation or a raising operation. Note that when the left working lever 41 is operated in the front-rear direction, the swing body 3 may perform the right swing operation or the left swing operation, and when the left working lever 41 is operated in the left-right direction, the arm 4B may perform the dumping operation or the excavation operation. The operation direction of the left working lever 41 and the operation direction of the right working lever 42 are arbitrarily related to the operation of the working equipment 4.
The left traveling pedal 43 and the right traveling pedal 44 are disposed on the front lower side of the operation seat 45. The left traveling pedal 43 is disposed to the left of the right traveling pedal 44. When the left traveling pedal 43 is operated, the crawler belt 2C on the left side of the traveling body 2 performs a forward operation or a backward operation. When the right traveling pedal 44 is operated, the crawler belt 2C on the right side of the traveling body 2 performs a forward operation or a backward operation.
In addition, in the remote operation room 200, a first monitor device 501 that displays work machine operation data indicating an operation status of the work machine 1 and an operation switch 502 that is operated to operate an electric device mounted on the work machine 1 are disposed. The first monitor device 501 displays, for example, a remaining amount of fuel of an engine mounted on the work machine 1, a temperature of coolant of the engine, a temperature of hydraulic oil for driving the hydraulic cylinder 5, and a traveling speed of the traveling body 2 as the work machine operation data. The operation switch 502 operates, for example, a headlight provided on the work machine 1 as an electric device mounted on the work machine 1.
The display device 50 displays the first image P1 and the second image P2 transmitted from the work machine 1. The first image P1 and the second image P2 are transmitted to the control device 60 of the remote operation system 100 via the control device 300 and the communication system 400. The control device 60 causes the display device 50 to display the first image P1 and the second image P2 acquired via the communication system 400. The display device 50 displays the second image P2 together with the first image P1. In the embodiment, the display device 50 displays the second image P2 so as to overlap a part of the first image P1.
The display device 50 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). In the embodiment, the display device 50 includes a plurality of flat panel displays disposed adjacent to each other. In the embodiment, the display device 50 includes a central display 51, a left display 52 disposed on a left side of the central display 51, a right display 53 disposed on a right side of the central display 51, an upper display 54 disposed above the central display 51, and a lower display 55 disposed below the central display 51.
The first image P1 displayed on the display device 50 is an image corresponding to the field of view of the front space of the operator when the operator is assumed to be seated on the driver's seat provided in the driver's cab 3A of the work machine 1. The operator in the remote operation room 200 can obtain a feeling of actually sitting on the driver's seat of the work machine 1.
The operator in the remote operation room 200 operates the remote operation device 40 to operate the working equipment 4, and excavates the work target. The excavated object excavated by the bucket 4C of the working equipment 4 is loaded onto a dump truck as a load.
In the embodiment, the dump truck is an unmanned dump truck that travels based on a control command transmitted from a control facility of the work site. In the remote operation room 200, a second monitor device 503 that displays dump truck operation data indicating an operation status of the unmanned dump truck at the work site is disposed. In the unmanned dump truck, a position sensor that detects position data of the unmanned dump truck is disposed. The position sensor detects an absolute position of the unmanned dump truck using a global navigation satellite system (GNSS). The second monitor device 503 displays the position of each of a plurality of unmanned dump trucks operating at the work site as the dump truck operation data. In addition, the operator can stop or depart the unmanned dump truck by operating an input device provided in the second monitor device 503.
In addition, in the remote operation room 200, a third monitor device 504 that displays guidance data of the working equipment 4 is disposed. As the guidance data, a relative distance between the target design surface of the work target and the working equipment 4, a shape of the work target, and an ore distribution of the work target are exemplified.
[Control Device]
The control device 300 includes a traveling body control unit 301, a swing body control unit 302, a working equipment control unit 303, an image data transmission unit 304, and a sensor data transmission unit 305.
The traveling body control unit 301 receives an operation signal of the remote operation device 40 transmitted from the control device 60. The traveling body control unit 301 outputs a control signal for controlling the operation of the traveling body 2 based on the operation signal of the remote operation device 40.
The swing body control unit 302 receives an operation signal of the remote operation device 40 transmitted from the control device 60. The swing body control unit 302 outputs a control signal for controlling the operation of the swing body 3 based on the operation signal of the remote operation device 40.
The working equipment control unit 303 receives an operation signal of the remote operation device 40 transmitted from the control device 60. The working equipment control unit 303 outputs a control signal for controlling the operation of the working equipment 4 based on the operation signal of the remote operation device 40. The control signal for controlling the working equipment 4 includes a control signal for controlling the hydraulic cylinder 5.
The image data transmission unit 304 transmits the image of the work site acquired by the imaging device 30 to the control device 60. The image data transmission unit 304 acquires the first image P1 of the first imaging range M1 from the first imaging device 31, and acquires the second image P2 of the second imaging range M2 from the second imaging device 32. The image data transmission unit 304 transmits the first image P1 and the second image P2 to the control device 60.
The sensor data transmission unit 305 transmits detection data of the sensor 70 mounted on the work machine 1 to the control device 60. In the embodiment, the sensor 70 includes the position sensor 71 that detects the position of the swing body 3, the posture sensor 72 that detects the posture of the swing body 3, and the angle sensor 73 that detects the angle of the working equipment 4. The position sensor 71 detects the absolute position of the swing body 3 using a global navigation satellite system (GNSS). The position sensor 71 includes a GNSS receiver provided in the swing body 3. The posture sensor 72 detects an inclination angle of the swing body 3 with respect to the horizontal plane. The posture sensor 72 includes an inertial measurement unit (IMU) provided in the swing body 3. The angle sensor 73 detects an angle of the working equipment 4. The angle sensor 73 includes a boom angle sensor that detects the angle of the boom 4A with respect to the swing body 3, an arm angle sensor that detects the angle of the arm 4B with respect to the boom 4A, and a bucket angle sensor that detects the angle of the bucket 4C with respect to the arm 4B.
The communication device 7 communicates with the communication device 6 via the communication system 400. The communication device 7 receives an operation signal of the remote operation device 40 transmitted from the control device 60 via the communication device 6, and outputs the operation signal to the control device 300. The communication device 7 transmits the first image P1 of the first imaging range M1 and the second image P2 of the second imaging range M2 received from the image data transmission unit 304 to the communication device 6 at the remote place. The communication device 7 includes an encoder that compresses the image data of the first image P1 and the image data of the second image P2. Each of the first image P1 and the second image P2 is transmitted from the communication device 7 to the communication device 6 in a compressed state.
The communication device 6 communicates with the communication device 7 via the communication system 400. The communication device 6 transmits an operation signal generated by operating the remote operation device 40 to the communication device 7. The communication device 6 receives the first image P1 and the second image P2 transmitted from the control device 300 via the communication device 7, and outputs the first image P1 and the second image P2 to the control device 60. The communication device 6 includes a decoder that restores the compressed image data of the first image P1 and the compressed image data of the second image P2. The first image P1 and the second image P2 are output from the communication device 6 to the control device 60 in a restored state.
The control device 60 includes an operation signal transmission unit 61, an image data reception unit 62, a first image processing unit 63, a second image processing unit 64, and a display control unit 65.
The operation signal transmission unit 61 transmits an operation signal for remotely operating the work machine 1. When the remote operation device 40 is operated by the operator, an operation signal for remotely operating the work machine 1 is generated. The operation signal transmission unit 61 transmits an operation signal of the remote operation device 40 to the control device 300.
The image data reception unit 62 receives the first image P1 and the second image P2. The image data reception unit 62 acquires the first image P1 and the second image P2 restored by the decoder of the communication device 6.
The first image processing unit 63 divides the first image P1 received by the image data reception unit 62.
The first image processing unit 63 divides the first image P1 into a plurality of images. The first image processing unit 63 divides the first image P1 into an image P11 to be displayed on the central display 51, an image P12 to be displayed on the left display 52, an image P13 to be displayed on the right display 53, an image P14 to be displayed on the upper display 54, and an image P15 to be displayed on the lower display 55.
The second image processing unit 64 generates a superimposed image P2S from the second image P2 received by the image data reception unit 62.
The second image processing unit 64 reduces the second image P2 to generate the superimposed image P2S.
The display control unit 65 causes the display device 50 to display the first image P1 and the second image P2. The display control unit 65 causes the display device 50 to display the second image P2 together with the first image P1. In the embodiment, the display control unit 65 causes the display device 50 to display the second image P2 together with the first image P1 such that the second image P2 overlaps a part of the first image P1.
The display control unit 65 superimposes the superimposed image P2S generated from the second image P2 on the lower portion of the first image P1 and causes the display device 50 to display the superimposed image P2S. In the embodiment, the display control unit 65 superimposes the superimposed image P2S on the image P15 and causes the lower display 55 to display the superimposed image P2S.
Note that, in the embodiment, the display control unit 65 causes the display device 50 to display a vehicle body data image P3 indicating the posture of the swing body 3, a working equipment data image P4 indicating the posture of the working equipment 4, a load data image P5 indicating the weight of the load to be loaded onto the dump truck, and a bucket data image P6 indicating the position of the tip blade 4D of the bucket 4C.
The display control unit 65 calculates the inclination angle of the swing body 3 with respect to the horizontal plane based on the detection data of the posture sensor 72. The display control unit 65 causes the display device 50 to display a symbol image indicating the inclination angle of the swing body 3 as the vehicle body data image P3. In the embodiment, the vehicle body data image P3 is displayed on the upper display 54.
In addition, the display control unit 65 calculates the posture of the working equipment 4 based on the detection data of the angle sensor 73. The display control unit 65 causes the display device 50 to display an animation image indicating the posture of the working equipment 4 as the working equipment data image P4. In the embodiment, the working equipment data image P4 is displayed on the right display 53.
In addition, the display control unit 65 calculates the weight of the load to be loaded onto the dump truck based on detection data of a weight sensor (not illustrated) that detects the weight of the load held by the bucket 4C. The display control unit 65 causes the display device 50 to display an indicator image indicating the weight of the load as the load data image P5. In the embodiment, the load data image P5 is displayed on the right display 53.
In addition, the display control unit 65 calculates the position of the tip blade 4D of the bucket 4C in the vertical direction based on the detection data of the angle sensor 73. The display control unit 65 causes the display device 50 to display an indicator image indicating the position in the vertical direction of the tip blade 4D of the bucket 4C as the bucket data image P6. In the embodiment, the bucket data image P6 is displayed on the right display 53.
[Remote Operation Method]
The first imaging device 31 images the first imaging range M1 including the front space SP1 of the swing body 3. The second imaging device 32 images the second imaging range M2 including the front lower space SP2 of the swing body 3. The image data transmission unit 304 transmits the first image P1 and the second image P2 to the control device 60 via the communication device 7 and the communication system 400. Each of the first image P1 and the second image P2 is transmitted from the work machine 1 to the control device 60 in a compressed state (step SA1).
The image data reception unit 62 receives the first image P1 and the second image P2 transmitted from the work machine 1 via the communication device 6. The image data reception unit 62 receives the restored first image P1 and second image P2.
The first image processing unit 63 divides the first image P1 into the image P11, the image P12, the image P13, the image P14, and the image P15 (step SB1).
The second image processing unit 64 generates the superimposed image P2S from the second image P2 (step SB2).
Note that the processing of step SB2 may be performed before the processing of step SB1, or the processing of step SB1 and the processing of step SB2 may be performed in parallel.
The display control unit 65 causes the display device 50 to display the first image P1. The display control unit 65 displays the image P11 on the central display 51, the image P12 on the left display 52, the image P13 on the right display 53, the image P14 on the upper display 54, and the image P15 on the lower display 55.
In addition, the display control unit 65 superimposes the second image P2 on a part of the first image P1 displayed on the display device 50. That is, the display device 50 displays the second image P2 so as to overlap a part of the first image P1. The second image P2 is superimposed on the lower portion of the first image P1. The superimposed image P2S of the second image P2 is displayed so as to be superimposed on the image P15 on the lower display 55 (step SB3).
[Method of Diagnosing Image Transmission Path]
Next, a method of diagnosing an image transmission path will be described. There is a possibility that an abnormality occurs in an image transmission path from a work machine to a remote place. Examples of the cause of the abnormality of the image transmission path include an abnormality of the image sensor of the imaging device 30, an abnormality of the communication device 7, an abnormality of at least a part of the communication system 400, and an abnormality of the communication device 6. The remote operation system 100 diagnoses the presence or absence of abnormality of the image transmission path. When there is an abnormality in the image transmission path, the remote operation system 100 allows the operator at the remote place to recognize the abnormality.
The diagnosis of the image transmission path is performed based on the first image P1 of the first imaging range M1 and the second image P2 of the second imaging range M2 in which an object whose state changes is included in an overlapping range MD between the first imaging range M1 and the second imaging range M2. In embodiments, the state of the object is a position of the object. In diagnosis of an image transmission path, an object whose position changes is used.
The object used to diagnose the image transmission path moves relative to the first imaging range M1 of the first imaging device 31 and the second imaging range M2 of the second imaging device 32. The diagnosis of the image transmission path is performed based on the first image P1 and the second image P2 including an object whose position changes in the overlapping range MD.
The first image P1 and the second image P2 used for diagnosis of the image transmission path are images captured at the same time.
In the embodiment, the object used for diagnosis of the image transmission path is the working equipment 4 attached to the swing body 3. When the remote operation device 40 is operated, an operation signal is transmitted from the operation signal transmission unit 61 at the remote place to the work machine 1. The working equipment 4 operates based on the operation signal transmitted from the remote place. The working equipment 4 moves relative to the first imaging range M1 and the second imaging range M2 based on the operation signal transmitted from the remote place.
At least a part of the working equipment 4 moves relative to the first imaging range M1 and the second imaging range M2 by the operation of the working equipment 4, and may be included in the overlapping range MD. As illustrated in
The communication device 7 of the work machine 1 transmits the first image P1 of the first imaging range M1 and the second image P2 of the second imaging range M2 including the bucket 4C whose position changes in the overlapping range MD between the first imaging range M1 and the second imaging range M2 to the control device 60 at the remote place.
The image data reception unit 62 of the control device 60 receives the first image P1 and the second image P2. The display control unit 65 causes the display device 50 to display the first image P1 and the second image P2 in which the bucket 4C whose position changes is included in the overlapping range MD between the first imaging range M1 and the second imaging range M2. The display control unit 65 causes the display device 50 to display the second image P2 together with the first image P1.
Each of
As illustrated in
As illustrated in
In addition, when the transmission path of the second image P2 is normal and the transmission path of the first image P1 is abnormal, the position of the bucket 4C changes in the second image P2, but the position of the bucket 4C does not change in the first image P1.
As described above, when the operator in the remote operation room 200 operates the remote operation device 40 to operate the working equipment 4, in a case where the position of the bucket 4C changes in each of the first image P1 and the second image P2 displayed on the display device 50, it is diagnosed that each of the transmission path of the first image P1 and the transmission path of the second image P2 is normal.
When the operator in the remote operation room 200 operates the remote operation device 40 to operate the working equipment 4, in a case where the position of the bucket 4C changes in the first image P1 displayed on the display device 50 but the position of the bucket 4C does not change in the second image P2, it is diagnosed that the transmission path of the first image P1 is normal and the transmission path of the second image P2 is abnormal.
When the operator in the remote operation room 200 operates the remote operation device 40 to operate the working equipment 4, in a case where the position of the bucket 4C changes in the second image P2 displayed on the display device 50 but the position of the bucket 4C does not change in the first image P1, it is diagnosed that the transmission path of the second image P2 is normal and the transmission path of the first image P1 is abnormal.
Note that the object used to diagnose the image transmission path may not be the working equipment 4. The object may be, for example, the ground GR. When the remote operation device 40 is operated, the traveling body 2 operates based on an operation signal transmitted from the remote place. When the traveling body 2 travels, the ground GR moves relative to the first imaging range M1 and the second imaging range M2.
When the operator in the remote operation room 200 operates the remote operation device 40 to operate the traveling body 2, in a case where the position of the ground GR changes in each of the first image P1 and the second image P2 displayed on the display device 50, it is diagnosed that each of the transmission path of the first image P1 and the transmission path of the second image P2 is normal.
When the operator in the remote operation room 200 operates the remote operation device 40 to operate the traveling body 2, in a case where the position of the ground GR changes in the first image P1 displayed on the display device 50 but the position of the ground GR does not change in the second image P2, it is diagnosed that the transmission path of the first image P1 is normal and the transmission path of the second image P2 is abnormal.
When the operator in the remote operation room 200 operates the remote operation device 40 to operate the traveling body 2, in a case where the position of the ground GR changes in the second image P2 displayed on the display device 50 but the position of the ground GRT does not change in the first image P1, it is diagnosed that the transmission path of the second image P2 is normal and the transmission path of the first image P1 is abnormal.
In addition, when the remote operation device 40 is operated, the swing body 3 operates based on an operation signal transmitted from the remote place. When the swing body 3 swings, the ground GR moves relative to the first imaging range M1 and the second imaging range M2. When the operator in the remote operation room 200 operates the remote operation device 40 to operate the swing body 3, the presence or absence of an abnormality in the transmission path of the first image P1 and the presence or absence of an abnormality in the transmission path of the second image P2 are diagnosed based on a change in the position of the ground GR in each of the first image P1 and the second image P2 displayed on the display device 50.
Note that the object used to diagnose the image transmission path may not be an object that moves relative to the first imaging range M1 and the second imaging range M2 based on the operation signal. For example, when a work machine different from the work machine 1 is present at the work site and at least a part of the different work machine is disposed in the overlapping range MD, the presence or absence of abnormality of the image transmission path is diagnosed based on the first image P1 and the second image P2 displayed on the display device 50. In addition, when there is an automobile traveling at the work site and at least a part of the automobile is disposed in the overlapping range MD, the presence or absence of abnormality of the transmission path of the first image P1 and the presence or absence of abnormality of the transmission path of the second image P2 are diagnosed based on the first image P1 and the second image P2 displayed on the display device 50.
Note that the change in the position of the object includes a change in the posture of the object. In the diagnosis of the image transmission path, an object whose posture changes may be used. For example, when the bucket 4C rotates about the bucket rotation axis CX, the posture of the bucket 4C changes. Depending on the relative positions of the first imaging device 31 and the second imaging device 32 and the bucket 4C, it may be difficult for the first imaging device 31 and the second imaging device 32 to grasp a change in the position of the bucket 4C, but the posture (rotation state) of the bucket 4C may be able to be grasped. In such a case, the object (bucket 4C) whose posture changes may be used in the diagnosis of the image transmission path.
Note that the state of the object may be the appearance of the object. For example, in a case where a blinking lamp is disposed in the overlapping range MD, if the transmission path of the first image P1 and the transmission path of the second image P2 are normal, the lamp blinks in each of the first image P1 and the second image P2. On the other hand, for example, when the transmission path of the first image P1 is normal and the image transmission path of the second image P2 is abnormal, the lamp blinks in the first image P1, but the lamp does not blink in the second image P2. When the transmission path of the second image P2 is normal and the transmission path of the first image P1 is abnormal, the lamp blinks in the second image P2, but the lamp does not blink in the first image P1. The presence or absence of an abnormality in the transmission path of the first image P1 and the presence or absence of an abnormality in the transmission path of the second image P2 are diagnosed based on the appearance (blinking state) of the lamp of each of the first image P1 and the second image P2 displayed on the display device 50.
[Computer System]
According to the above-described embodiment, the computer program or the computer system 1000 can execute: receiving the first image P1 of the first imaging range M1 around the work machine 1 and the second image P2 of the second imaging range M2 at least partially overlapping the first imaging range M1; and displaying the first image P1 and the second image P2 when the object whose state changes is disposed in the overlapping range MD between the first imaging range M1 and the second imaging range M2 on the display device 50.
[Effects]
As described above, according to the embodiment, the first image P1 and the second image P2 in which the object whose state changes is included in the overlapping range MD between the first imaging range M1 and the second imaging range M2 are displayed on the display device 50. The operator in the remote operation room 200 can recognize the presence or absence of an abnormality in the image transmission path based on the state of the object of each of the first image P1 and the second image P2 displayed on the display device 50.
Since the object used for diagnosis of the image transmission path is an object that operates based on the operation signal, the operator in the remote operation room 200 can change the state of the object by operating the remote operation device 40. The operator in the remote operation room 200 can diagnose the image transmission path at an arbitrary timing. The operator in the remote operation room 200 can diagnose the image transmission path, for example, at the timing of the start of work inspection.
Since the object used to diagnose the image transmission path is the working equipment 4, the operator in the remote operation room 200 can diagnose the image transmission path with high frequency. When the excavation work of the work target is performed, the working equipment 4 is operated at a high frequency. Therefore, since the object used to diagnose the image transmission path is the working equipment 4, the operator in the remote operation room 200 can diagnose the image transmission path with high frequency.
In the embodiment described above, the first image P1 and the second image P2 including an object are displayed on the display device 50, and the operator in the remote operation room 200 diagnoses the presence or absence of abnormality in the image transmission path based on the state of each object of the first image P1 and the second image P2 displayed on the display device 50. The control device 60 may diagnose the presence or absence of abnormality of the image transmission path. The control device 60 performs image processing on each of the first image P1 and the second image P2 transmitted from the work machine 1, and extracts an object from each of the first image P1 and the second image P2. The control device 60 may compare the state of the object of the first image P1 with the state of the object of the second image P2 to diagnose the presence or absence of abnormality of the image transmission path.
In the above-described embodiment, the second imaging device 32 is disposed below the first imaging device 31 in the vertical direction of the swing body 3. In the vertical direction of the swing body 3, the second imaging device 32 may be disposed above the first imaging device 31, or may be disposed at the same position as the first imaging device 31.
In the above-described embodiment, the first imaging device 31 is disposed at the same position as the second imaging device 32 in the left-right direction of the swing body 3. In the left-right direction of the swing body 3, the first imaging device 31 may be disposed on the left or right of the second imaging device 32.
In the above-described embodiment, the first imaging device 31 is disposed behind the second imaging device 32 in the front-rear direction of the swing body 3. In the front-rear direction of the swing body 3, the first imaging device 31 may be disposed in front of the second imaging device 32, or may be disposed at the same position as the second imaging device 32.
In the above-described embodiment, the first imaging device 31 and the second imaging device 32 may be disposed adjacent to each other in the vertical direction or the left-right direction, and may be disposed such that the angle of the optical axis OA1 of the first imaging device 31 is different from the angle of the optical axis OA2 of the second imaging device 32 in the vertical direction. In addition, the first imaging device 31 and the second imaging device 32 may be disposed adjacent to each other in the vertical direction or the left-right direction, and may be disposed such that the angle of the optical axis OA1 of the first imaging device 31 is different from the angle of the optical axis OA2 of the second imaging device 32 in the left-right direction.
In the above-described embodiment, the superimposed image P2S of the second image P2 is displayed so as to overlap the lower portion of the first image P1. The superimposed image P2S may be displayed so as to overlap the upper portion of the first image P1, may be displayed so as to overlap the left portion of the first image P1, or may be displayed so as to overlap the right portion of the first image P1.
In the above-described embodiment, the first image P1 and the second image P2 may not overlap each other. The first image P1 and the second image P2 may be displayed side by side on the display device 50.
In the above-described embodiment, the first imaging device 31 may be disposed outside the driver's cab 3A. The first imaging device 31 may be disposed on the swing body 3.
As described above, an unmanned dump truck may operate at a work site. The unmanned dump truck travels based on a control command transmitted from a control facility of the work site. In the above-described embodiment, the image captured by the imaging device 30 may be transmitted to the control facility of the work site and displayed on the display device installed in the control facility. The control facility is disposed at a remote place of the work machine 1. The manager of the work site present in the control facility can see the image captured by the imaging device 30.
In the above-described embodiment, the swing body 3 (vehicle body) includes the driver's cab 3A. The swing body 3 may not include the driver's cab 3A. In a case where the driver's cab 3A does not exist in the swing body 3, the first imaging device 31 is disposed at the front portion of the swing body 3.
In the above-described embodiment, the display device 50 includes the plurality of displays (51, 52, 53, 54, and 55). The display device 50 may include one display. In addition, the display may be a flat panel display or a curved panel display.
Note that, in the embodiment described above, the work machine 1 is a loading excavator. The work machine 1 may be a backhoe. In addition, in the embodiment described above, the work machine 1 includes the swing body 3 as the vehicle body, but the vehicle body of the work machine 1 may not be the swing body. The work machine 1 only needs to be a work machine having working equipment, and may be a bulldozer or a wheel loader.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-028617 | Feb 2020 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2021/002449 | 1/25/2021 | WO |
