IMAGE DISPLAY SYSTEM, IMAGE DISPLAY COMPOSITE SYSTEM, AND IMAGE DISPLAY METHOD

TECHNICAL FIELD

The present invention relates to an image display system that causes an output interface of an information terminal apparatus to output specification information about interactions between work machines and operators.

BACKGROUND ART

There has been proposed a technology of displaying working information about a construction machine, such as an engine start date and time, an engine stop date and time, an engine cooling water amount, an engine oil amount, a hydraulic oil amount, a filter differential pressure, and a work position, on a monitor screen of a management computer capable of communicating with the construction machine (see, for example, Patent Literature 1).

There has been proposed a management system that superimposes a symbol indicating a position of each industrial vehicle on a map and displays the symbol in a different aspect according to the machine number of the industrial vehicle, location information, the vehicle ID, time information by an hour meter, a maintenance cost amount, and a working situation such as the cumulative number of warnings (see, for example, Patent Literature 2).

Further, it has been conventionally performed to present work results of an operator at the time of operating a work machine to the operator in order to improve the work efficiency and skill level of the operator.

For example, a work machine operation support apparatus that calculates work results indicating efficiency of work based on a work amount, work time, and power consumption of a work machine is disclosed in Patent Literature 3.

CITATION LIST
Patent Literature

- Patent Literature 1: Japanese Patent Laid-Open No. 2002-088821
- Patent Literature 2: Japanese Patent Laid-Open No. 2013-235485
- Patent Literature 3: Japanese Patent Laid-Open No. 2016-141940

SUMMARY OF INVENTION
Technical Problem

The management system in the conventional technology assumes a case where a work machine is operated by an operator who is on board the work machine (an actual machine operation) but does not assume a case where a work machine is operated by an operator in a remote control room installed in a place that is remote from a specified area (a remote operation) and a case where a work machine is automatically operated are not assumed.

Further, though, in Patent Literature 3, work results of an operator who is on board one work machine are acquired, it is difficult to acquire appropriate work results, for example, in a case where the same operator transfers from one work machine to another work machine to perform work.

Therefore, an object of the present invention is to provide a system capable of causing a user to know the state of each of one or more work machines working in various forms in a specified area, and the like. Further, another object is to provide an image display system capable of displaying a work history of each work machine for each operator who operates the work machine, an image display composite system, and an image display method.

Solution to Problem

An image display system of the present invention causes an output interface of an information terminal apparatus to output specification information about interactions between the work machines and operators.

According to the image display system with the above configuration, it is possible to cause a user of the information terminal apparatus to know the specification information about the interactions between the work machines and the operators on the output interface of the information terminal apparatus.

In the image display system with the above configuration, it is preferable that the image display system causes the output interface of the information terminal apparatus to output indicator images associated with positions of the work machines existing in a specified area in a specified area image, the indicator images identifiably indicating current operation statuses of the work machines among a plurality of selectable or interchangeable operation statuses of the work machines, as the specification information.

In the image display system with the above configuration, it is preferable to cause the output interface of the information terminal apparatus to output pieces of identification information about the operators of the work machines and pieces of operation history information about the work machines of the operators, as the specification information.

According to the image display system with the above configuration, it is possible to cause the user of the information terminal apparatus to know, through the pieces of identification information about the operators and the pieces of operation history information about the work machines of the operators, the pieces of operation history information associated with the operators.

An image display composite system of the present invention comprises: the image display system described above, and either the information terminal apparatuses, or the work machines and a remote operation apparatus for remotely operating the work machines.

An image display method of the present invention comprises a process of causing an output interface of an information terminal apparatus to output indicator images associated with positions of work machines existing in a specified area in a specified area image, the indicator images identifiably indicating current operation statuses of the work machines among a plurality of selectable or interchangeable operation statuses of the work machines.

An image display system of the present invention comprises a process of causing an output interface of an information terminal apparatus to output pieces of identification information about operators of work machines and pieces of operation history information about the work machines of the operators.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram about configurations of an image display composite system and an image display system as a first embodiment of the present invention.

FIG. 2 is a diagram about a configuration of a remote operation apparatus as the first embodiment of the present invention.

FIG. 3 is a diagram about a configuration of a work machine as the first embodiment of the present invention.

FIG. 4 is a diagram about functions of the image display system as the first embodiment of the present invention.

FIG. 5 is a diagram about a specified area image and a first display form of indicator images as the first embodiment of the present invention.

FIG. 6A is a diagram about a second display form of each indicator image as the first embodiment of the present invention.

FIG. 6B is a diagram about a third display form of each indicator image as the first embodiment of the present invention.

FIG. 6C is a diagram about a fourth display form of each indicator image as the first embodiment of the present invention.

FIG. 7 is a diagram about one display form of actual-machine-related information in the specified area image as the first embodiment of the present invention.

FIG. 8 is a diagram about one display form of weather information in the specified area image as the first embodiment of the present invention.

FIG. 9 is a diagram about one display form of notification information in the specified area image as the first embodiment of the present invention.

FIG. 10 is a diagram about configurations of an image display composite system and an image display system as a second embodiment of the present invention.

FIG. 11 is a diagram about a configuration of a remote operation apparatus as the second embodiment of the present invention.

FIG. 12 is a diagram about a configuration of a work machine as the second embodiment of the present invention.

FIG. 13 is a diagram about functions of the image display system as the second embodiment of the present invention.

FIG. 14 is a diagram showing a display form of an information terminal apparatus as the second embodiment of the present invention.

FIG. 15 is a diagram showing another display form in the second embodiment of the present invention.

FIG. 16 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 17 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 18 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 19 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 20 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 21 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 22 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 23 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 24 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 25 is a diagram showing still another display form in the second embodiment of the present invention.

FIG. 26 is a diagram showing a data structure of operation history information as the second embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS
First Embodiment

The following description is description about an image display composite system as a first embodiment of the present invention.

(Configuration of Image Display Composite System)

The image display composite system shown in FIG. 1 is configured with an image display system 10, a remote operation apparatus 20, and/or a work machine 40 which is a target of remote operation by the remote operation apparatus 20. The image display system 10, the remote operation apparatus 20, and the work machine 40 are configured to be capable of mutually performing network communication. A mutual communication network between the image display system 10 and the remote operation apparatus 20 and a mutual communication network between the image display system 10 and the work machine 40 may be the same or different.

(Configuration of Image Display System)

The image display system 10 is configured with a computer existing separately from the remote operation apparatus 20 and the work machine 40, and comprises a database 102, a first functional element 121 and a second functional element 122.

The database 102 may be configured with a database server capable of mutually communicating with the image display system 10. With the database 102, operator identifiers for identifying operators of work machines 40, pieces of particular information about and/or attributes of the operators are registered, being mutually associated. The particular information about each operator includes, for example, a face picture image, face illustration image, and/or name of the operator. The attributes of each operator include, for example, an organization that the operator belongs to, a language used by the operator, and/or a work job history of the operator.

The particular information about and/or attributes of each operator who is remotely operating a work machine 40 through the remote operation apparatus 20 is transmitted from the remote operation apparatus 20 or the work machine 40 to the image display system 10 and then registered with the database 102, being mutually associated. The particular information about and/or attributes of the operator, and the operator identifier may be inputted by the operator through a touch panel constituting a remote input interface 210 or may be read from an electromagnetic storage medium such as a card or mobile information terminal apparatus owned by the operator by a contactless reader constituting the remote input interface 210.

With the database 102, an actual-machine identifier for identifying each work machine 40 and related information about the work machine 40 are registered in association with each other. The related information about the work machine 40 includes, in addition to static actual-machine-related information, such as the class and specifications of the work machine 40, the type of an attachment 445, and the type of crawlers constituting a lower traveling body 410, dynamic actual-machine-related information, such as a place where the remote operation apparatus 20 exists if the work machine 40 is remotely operated, cumulative operation time, a remaining amount of fuel, a remaining amount of soot of an exhaust gas purification device, or a remaining amount of urea water per unit period such as one day or one month.

The actual-machine identifier and the static actual-machine-related information are transmitted from each work machine 40 to the image display system 10 and then registered with the database 102 in association with each other. The actual-machine identifier and the dynamic actual-machine-related information are transmitted from each work machine 40 to the image display system 10 and then cumulatively or sequentially registered with the database 102 in association with each other. The dynamic actual-machine-related information is measured using appropriate sensors constituting an actual-machine sensor group 416 mounted on the work machine 40. The dynamic actual-machine-related information may include internal state variables and/or external state variables of the work machine 40. The “external state variables” are variables indicating an environment of the work machine 40, which influences the internal state variables of the work machine 40. For example, an ambient temperature around the work machine 40 is an external state variable. The “internal state variables” are variables indicating states of components of the work machine 40, for example, a water temperature in piping for engine cooling water, a temperature of hydraulic piping, a tank, and the like of a hydraulic system for causing a work mechanism 440 and the like to operate.

The related information about each work machine 40 is transmitted to the image display system 10 through the remote operation apparatus 20 and then registered with the database 102 in association with its actual-machine identifier.

The particular information about and/or attributes of each operator who is actually operating a work machine 40 may be transmitted to the image display system 10 and then registered with the database 102 in association with an actual-machine identifier. The particular information about and/or attributes of the operator may be inputted by the operator through a touch panel constituting an actual-machine input interface 41 or may be read from an electromagnetic storage medium such as a card or a mobile terminal owned by the operator by a contactless reader constituting the actual-machine input interface 41.

Each functional element is configured with an arithmetic processing unit (for example, a single-core processor and/or a multi-core processor, or a processor core constituting the multi-core processor), and the functional element reads necessary data and software from a storage device such as a memory and executes arithmetic processing described later according to the software, for the data.

(Configuration of Remote Operation Apparatus)

Each remote operation apparatus 20 comprises a remote control device 200, the remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing unit (for example, a single-core processor and/or a multi-core processor, or a processor core constituting the multi-core processor), and the remote control device 200 reads necessary data and software from a storage device such as a memory and executes arithmetic processing according to the software, for the data.

The remote input interface 210 comprises a remote operation mechanism 211. The remote output interface 220 comprises a remote image output device 221 and a remote wireless communication equipment 222 for performing communication via a network.

The remote operation mechanism 211 includes a travel operation device, a swing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each of the operation devices has an operation lever to receive a rotation operation. The operation lever of the travel operation device (a travel lever) is operated to move the lower traveling body 410 of the work machine 40. The travel lever may serve as a travel pedal. For example, a travel pedal fixed to a base part or lower end part of the travel lever may be provided. The operation lever of the swing operation device (a swing lever) is operated to move a hydraulic swing motor constituting a swing mechanism 430 of the work machine 40. The operation lever of the boom operation device (a boom lever) is operated to move a boom cylinder 442 of the work machine 40. The operation lever of the arm operation device (an arm lever) is operated to move an arm cylinder 444 of the work machine 40. The operation lever of the bucket operation device (a bucket lever) is operated to move a bucket cylinder 446 of the work machine 40.

Each of the operation levers constituting the remote operation mechanism 211 are arranged, for example, around a seat St for an operator to sit on, as shown in FIG. 2. Though the seat St is in a form like a high-back chair with arm rests, it may be a seating portion in any form where an operator can be seated, such as a form of a low-back chair without a head rest or a chair without a backrest.

In front of the seat St, a pair of left and right travel levers 2110 corresponding to left and right crawlers are arranged side by side left and right. One operation lever may serve as a plurality of operation levers. For example, a left-side operation lever 2111 provided at the front of a left-side frame of the seat St shown in FIG. 2 may function as the arm lever when being operated forward and backward and function as the swing lever when being operated in left and right directions. Similarly, a right-side operation lever 2112 provided at the front of a right-side frame of the seat St shown in FIG. 2 may function as the boom lever when being operated forward and backward and function as the bucket lever when being operated in left and right directions. Lever patterns may be freely changed by operation instructions by an operator.

For example, as shown in FIG. 2, the remote image output device 221 is configured with a central remote image output device 2210, a left-side remote image output device 2211, and a right-side remote image output device 2212 that are arranged in front of, diagonally forward to the left of, and diagonally forward to the right of the seat St, each of which having an approximately rectangular screen. The screens (image display areas) of the central remote image output device 2210, the left-side remote image output device 2211, and the right-side remote image output device 2212 may be the same or different in shape and size.

As shown in FIG. 2, the right edge of the left-side remote image output device 2211 adjoins the left edge of the central remote image output device 2210 such that the screen of the central remote image output device 2210 and the screen of the left-side remote image output device 2211 form an inclination angle of θ1 (for example, 120°≤θ1≤150°). As shown in FIG. 2, the left edge of the right-side remote image output device 2212 adjoins the right edge of the central remote image output device 2210 such that the screen of the central remote image output device 2210 and the screen of the right-side remote image output device 2212 form an inclination angle of θ2 (for example, 120°≤θ2≤150°). The inclination angles θ1 and θ2 may be the same or different.

The screen of each of the central remote image output device 2210, the left-side remote image output device 2211, and the right-side remote image output device 2212 may be parallel or inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left-side remote image output device 2211, and the right-side remote image output device 2212 may be configured with a plurality of divided remote image output devices. For example, the central remote image output device 2210 may be configured with a pair of vertically adjoining remote image output devices each of which has an approximately rectangular screen.

The remote image output device 221 may be configured with a single remote image output device that is curved or bent, surrounding the seat St. A single remote image output device may be configured, for example, with the central remote image output device 2210. The remote image output device 221 may be configured with two remote image output devices (for example, the central remote image output device 2210, and the left-side remote image output device 2211 or the right-side remote image output device 2212).

(Configuration of Work Machine)

Each work machine 40 comprises an actual-machine control device 400, the actual-machine input interface 41, an actual-machine output interface 42, and the work mechanism 440. The remote control device 400 is configured with an arithmetic processing unit (a single-core processor, or a multi-core processor or a processor core constituting the multi-core processor), and the remote control device 400 reads necessary data and software from a storage device such as a memory and executes arithmetic processing according to the software, for the data.

The work machine 40 is, for example, a crawler shovel (a construction machine) and comprises the crawler-type lower traveling body 410 and an upper swinging body 420 that is swingably mounted on the lower traveling body 410 via the swinging mechanism 430 as shown in FIG. 3. On a front left-side part of the upper swinging body 420, a cab 424 (an operation room) is provided. On a front central part of the upper swinging body 420, the work mechanism 440 is provided.

The actual-machine input interface 41 comprises an actual-machine operation mechanism 411, an actual-machine image-pickup device 412, an actual-machine positioning device 414, and the actual-machine sensor group 416. The actual-machine operation mechanism 411 comprises a plurality of operation levers arranged similarly to those of the remote operation mechanism 211, around a seat arranged inside the cab 424. A drive mechanism or a robot that receives a signal corresponding to an operation aspect of a remote operation lever and moves an actual-machine operation lever based on the received signal is provided in the cab 424. The actual-machine image-pickup device 412 is installed, for example, inside the cab 424 and picks up an image of an environment including at least a part of the work mechanism 440 (for example, the attachment 445) through a front window. A part or all of the front window and side windows may be omitted. The actual-machine positioning device 414 is a device for measuring a current position of the work machine 40 and is configured, for example, a GNSS (Global Navigation Satellite System), and a gyro sensor if necessary. The actual-machine sensor group 416 is configured with various sensors for measuring the situation of the work machine 40 including a remaining amount of fuel of the work machine 40 and the like.

The actual-machine output interface 42 comprises an actual-machine wireless communication equipment 422 for performing communication via a network.

As shown in FIG. 3, the work mechanism 440 as a work mechanism comprises a boom 441 fitted to the upper swinging body 420 in a manner that it can be raised and lowered, an arm 443 rotatably coupled with the tip of the boom 441, and the attachment 445 (for example, a bucket) rotatably coupled with the tip of the arm 443. The boom cylinder 442, the arm cylinder 444 and the bucket cylinder 446, which are configured with extendable hydraulic cylinders, are fitted to the work mechanism 440.

The boom cylinder 442 intervenes between the boom 441 and the upper swinging body 420 so as to, by receiving supply of hydraulic oil, extend or contract to cause the boom 441 to rotate in a raising or lowering direction. The arm cylinder 444 intervenes between the arm 443 and the boom 441 so as to, by receiving supply of hydraulic oil, extend or contract to cause the arm 443 to rotate around a horizontal axis relative to the boom 441. The bucket cylinder 446 intervenes between the attachment 445 and the arm 443 so as to, by receiving supply of hydraulic oil, extend or contract to cause the attachment 445 to rotate around a horizontal axis relative to the arm 443.

(Configuration of Information Terminal Apparatus)

An information terminal apparatus 60 is a terminal apparatus such as a smartphone, a tablet terminal, or a personal computer and comprises a control device 600, an input interface 610, and an output interface 620. The control device 600 is configured with an arithmetic processing unit (a single-core processor, or a multi-core processor or a processor core constituting the multi-core processor), and the control device 600 reads necessary data and software from a storage device such as a memory and executes arithmetic processing according to the software, for the data.

The input interface 610 is configured with a touch-panel buttons, switches, and the like. The remote output interface 620 comprises a remote image output device and wireless communication equipment for performing communication via a network.

(Functions)

FIG. 4 shows a flowchart showing the image display system configured as above and functions of the image display system. In the flowchart, each block indicated by “C˜” is used for simplification of description, means transmission and/or reception of data, and means such a condition branch that processing in a branch direction is executed subject to the transmission and/or reception of the data.

In the information terminal apparatus 60, it is determined by the control device 600 whether a first specification operation through the input interface 610 has been made or not (FIG. 4/STEP610). The “first specification operation” includes, for example, an operation of tapping a keyboard and/or a menu choice through the input interface 610 for specifying factors to identify a specified area, and the like.

If a result of the determination is negative (FIG. 4/STEP610: NO), the process of and after the determination about whether the first specification operation has been made or not is repeated. On the other hand, if the result of the determination is positive (FIG. 4/STEP610: YES), a specified area information request is transmitted from the information terminal apparatus 60 to the image display system 10 by the control device 600 (FIG. 4/STEP612). The request includes, for example, identification factors for identifying a specified area. The identification factors may include the name of the specified area, the name or address of the location thereof, or the latitude and longitude, or a latitude/longitude range of a representative spot, or the like. The identification factors may include actual-machine identifiers for identifying work machines 40 existing in the specified area, operator identifiers for identifying operators operating the work machines 40, and/or remote identifiers for identifying remote operation apparatus 20 operating or scheduled to operate the work machines 40.

In the image display system 10, when the specified area information request is received (FIG. 4/C11), a specified area image showing a situation of the specified area identified by the identification factors included in the request is acquired by the first functional element 121 (FIG. 4/STEP110). Thereby, for example, as shown in FIG. 5, a bird's-eye-view image of the specified area in which eight work machines Q1 to Q8 are included, is acquired.

The specified area image is acquired through an image pickup apparatus mounted on an airplane (for example, an unmanned airplane) flying above the specified area or an artificial satellite, an image pickup apparatus installed in the specified area, and/or actual-machine image-pickup devices 412 mounted on work machines 40 existing in the specified area. The picked-up images may be accumulated in the database 102 and/or an external database and retrieved by the first functional element 121.

In the image display system 10, the position (the actual-machine positions) and identifier (actual-machine identifier) of each work machine 40 existing in the specified area shown by the specified area image are recognized by the first functional element 121 (FIG. 4/STEP112).

Specifically, in the work machine 40, the actual-machine position (the latitude and longitude) of the work machine 40 in the actual space is recognized through the actual-machine positioning device 414 by the actual-machine control device 400. The actual-machine position or actual-machine position data indicating the actual-machine position is transmitted to the image display system 10 together with the actual-machine identifier through the actual-machine wireless communication equipment 422 by the actual-machine control device 400 and then registered with the database 102 in association with each other. The actual-machine position of the work machine 40 included in the latitude/longitude range indicating the specified area, and the actual-machine identifier associated with the actual-machine position are retrieved from the database 102 by the first functional element 121. The actual-machine position (pixel positions) in a specified area image coordinate system by the first functional element 121 is recognized based on the actual-machine position in an actual space coordinate system.

Thereby, for example, in addition to the work machines Q1 to Q8 included in the specified area image as shown in FIG. 5, the actual-machine position and actual-machine identifier of a work machine 40 existing in the specified area are recognized even if the work machine 40 is not in the specified area image because it is hidden behind a structure or the like.

In the image display system 10, the current operation status of each work machine 40 existing in the specified area shown by the specified area image, among a plurality of selectable or changeable operation statuses, is recognized by the first functional element 121 (FIG. 4/STEP114).

Specifically, in the work machine 40, the current operation status is recognized by the actual-machine control device 400. The plurality of operation statuses includes a “remote operation status” in which the work machine 40 is remotely operated by an operator through the remote operation apparatus 20, an “actual-machine operation status” in which the work machine 40 is operated by the operator who is on board the cab 424, and a “no operation status” in which the work machine 40 is not being operated. The no operation status may include, in addition to the status of the work machine 40 being stopped, a status of being automatically operated according to software.

When communication between the work machine 40 and the remote operation apparatus 20 is established, a remote operation command corresponding to the operation aspect of the remote operation mechanism 211 is transmitted from the remote operation apparatus 20 to the actual-machine control device 400, and operation and the like of the work mechanism 440 are being controlled according to the remote operation command, it is recognized that the current operation status is the “remote operation status.” In this case, the operator identifier of an operator who is operating the remote operation apparatus 20 is recognized. At the time of performing a remote operation of the work machine 40, a picked-up image acquired through the actual-machine image-pickup device 412 or a model image generated according thereto is outputted to the remote image output device 221 of the remote operation apparatus 20.

When communication between the work machine 40 and the remote operation apparatus 20 has not been established, an actual-machine operation command corresponding to the operation aspect of the actual-machine operation mechanism 411 is transmitted to the actual-machine control device 400, and operation and the like of the work mechanism 440 are being controlled according to the actual-machine operation command, it is recognized that the current operation status is the “actual-machine operation status.” In this case, the operator identifier of an operator who is operating the work machine is recognized.

When the operation command has not been transmitted to the actual-machine control device 400, it is recognized that the current operation status is the “no operation status.”

The current operation status or current operation status data indicating the current operation status is transmitted to the image display system 10 together with the actual-machine identifier and/or the operator identifier through the actual-machine wireless communication equipment 422 by the actual-machine control device 400 and then registered with the database 102 being mutually associated. Then, the current operation status of the work machine 40 associated with the actual-machine identifier is retrieved from the database 102 by the first functional element 121.

In the image display system 10, specified area image and indicator image output commands are generated by the second functional element 122 and then transmitted to the information terminal apparatus 60 (FIG. 4/STEP116). The specified area image output command and the indicator image output command may be transmitted to the information terminal apparatus 60 not at the same timing but at different timings.

In the information terminal apparatus 60, when the indicator image output command is received (FIG. 4/C21), the indicator images are outputted to the remote image output device constituting the output interface 620 by the control device 600, as specification information (FIG. 4/STEP614). The “indicator images” are images that are associated with the positions of the work machines 40 existing in the specified area and identifiably indicates the current operation statuses of the work machines 40. The conception of “being associated with the positions” is a conception that encompasses all such forms that relations to the work machines 40 existing at the positions can be inferred, such as being arranged at or near the positions and being connected to the positions by lines or the like.

Thereby, for example, in the specified area image shown in FIG. 5, face picture images M11, M12, and M15 are outputted as indicator images near images indicating the work machines Q1, Q2, and Q5, respectively. The face picture images M11, M12, and M15 as indicator images are face picture images of operators remotely operating the work machines Q1, Q2, and Q5, respectively. The face picture images of the operators are retrieved from the database 102, for example, based on actual-machine identifiers and/or operator identifiers. In a specified area image, a face picture image as an indicator image that is outputted in association with the position of a work machine 40 indicates that the current operation status of the work machine 40 is the remote operation status (one first operation status). In addition to particular information about the operators, attributes of the operators may be identifiably indicated by the indicator images M1i.

In the specified area image shown in FIG. 5, face illustration images (or icons imitating a human figure) M13, M14, M17, and M18 are outputted near images indicating the work machines Q3, Q4, Q7, and Q8, respectively, as indicator images. The face illustration images M13, M14, M17, and M18 as the indicator images are face illustration images of operators remotely operating the work machines Q3, Q4, Q7, and Q8, respectively. The face illustration images of the operators are retrieved from the database 102, for example, based on actual-machine identifiers and/or operator identifiers. In a specified area image, a face illustration image as an indicator image that is outputted in association with the position of a work machine 40 indicates that the current operation status of the work machine 40 is the actual-machine operation status (another first operation status). When a work machine 40 is automatically operated, an image indicating automatic operation (for example, an icon image imitating a humanoid robot) or a face illustration image of a person who has given an automatic operation instruction is displayed as the face illustration image as an indicator image.

In the specified area image shown in FIG. 5, neither a face picture image nor a face illustration image is displayed near the image indicating the work machine Q6. An indicator image not being outputted in association with the position of a work machine 40 indicates that the current operation status of the work machine 40 is the no operation status (a second operation status).

As shown in FIG. 6A, in addition to or instead of the face picture images and the face illustration images, approximately circular indicator images M2i may be displayed around the work machines Qi (i=1, 2, . . . ) or the positions of the work machines Qi may be displayed in the specified area image. For example, the current operation statuses of the work machines Qi are identifiably indicated by different colors and/or blinking patterns of the indicator images M2i. The current operation status may include a status of communication between a work machine 40 and the remote operation apparatus 20, specifically, a communication established status and a communication unestablished status, and/or a status of a work machine 40 waiting in the remote operation status and the communication established status.

In addition, as shown in FIG. 6B, an approximately triangular indicator image M3i may be displayed near the position of a work machine Qi in the specified area image. For example, according to whether the indicator image M3i is displayed or not, whether there is an abnormality in a corresponding work machine Qi or not is identifiably indicated. If there is some abnormality in the work machine Qi, (example: an abnormality of the actual-machine sensor group 416), the indicator image M3i is displayed.

If the abnormality of the work machine Qi is a failure in communication with the remote operation apparatus 20 of the work machine 40 (example: decrease in communication speed, decrease in communication speed stability), an indicator image M4i configured with a plurality of circular arcs and a×sign may be displayed near the position of the work machine Qi as shown in FIG. 6C. For example, according to whether the indicator image M4i is displayed or not, it is identifiably indicated whether there is a communication failure in the work machine Qi or not. If there is a communication failure in the work machine Qi, the indicator image M4i is displayed.

In the information terminal apparatus 60, it is determined by the control device 600 whether a second specification operation through the input interface 610 has been made on any indicator image M1i in the specified area image or not (FIG. 4/STEP616). The “second specification operation” includes, for example, an operation of tapping on a specified position, such as the indicator image M1i, through the input interface 610.

If a result of the determination is negative (FIG. 4/STEP616: NO), the process of and after the determination about whether the first specification operation has been made or not is repeated. On the other hand, if the result of the determination is positive (FIG. 4/STEP616: YES), an actual-machine-related information request is transmitted from the information terminal apparatus 60 to the image display system 10 by the control device 600 (FIG. 4/STEP618). The request includes, for example, an operator identifier and/or an actual-machine identifier.

In the image display system 10, when the actual-machine-related information request is received (FIG. 4/C12), actual-machine-related information related to a work machine 40 and/or an operator of the work machine 40 identified by the identifier/identifiers included in the request is recognized by the first functional element 121 (FIG. 4/STEP118). For example, the actual-machine-related information is retrieved from the database 102 based on the actual-machine identifier.

In the image display system 10, the actual-machine-related information or a command to output the information is transmitted to the information terminal apparatus 60 by the second functional element 122 (FIG. 4/STEP120).

In the information terminal apparatus 60, when the actual-machine-related information or the command to output the information is received (FIG. 4/C22), the actual-machine-related information is outputted to the remote image output device constituting the output interface 620 by the control device 600 (FIG. 4/STEP620). The actual-machine-related information includes, in addition to the static actual-machine-related information, such as the class and specifications of the work machine 40, the type of the attachment 445, and the type of the crawlers constituting the lower traveling body 410, the dynamic actual-machine-related information, such as a place where the remote operation apparatus 20 exists if the work machine 40 is remotely operated, cumulative operation time, a remaining amount of fuel, a remaining amount of soot of the exhaust gas purification device, or a remaining amount of urea water per unit period such as one day or one month.

Thereby, for example, as shown in FIG. 7, a window PM showing the actual-machine-related information is displayed in association with the indicator image M15 associated with the position of the work machine Q5 in the specified area image. In a window PM, the model number of a work machine 40, the name and icon of an operator OP who is on board the work machine Q5, the remaining fuel of the work machine 40, the remaining amount of urea water of the work machine 40, working time after the last maintenance of the work machine 40, time at the work site, a map of the work site, and the like are displayed. By a tap operation of a specification icon included in the window PM being performed, a call function and/or a chat function between a user of the information terminal apparatus 60 and the operator of the work machine 40 may be activated through the remote operation apparatus 20 and/or the work machine 40, through the information terminal apparatus 60.

Advantageous Effects

According to the image display system 10 with the above configuration, on the output interface 620 of the information terminal apparatus 60, face picture images and face illustration images as the indicator images M1i, which are associated with the positions of work machines Qi existing in a specified area in a specified area image, are displayed as specification information (see FIG. 4/STEP610→STEP612→ . . . →STEP110→STEP112→STEP114→STEP116→ . . . →STEP 614, and FIG. 5). It is possible to, according to difference among the indicator images M1i in type and/or design (including color, shape, pattern, or a combination thereof, or a dynamic change aspect of the design such as a blinking pattern), cause the user of the information terminal apparatus 60 to know whether the current operation status is the remote operation status or the actual-machine operation status, among a plurality of selectable operation statuses of the work machines Qi. The face picture image and the face illustration image correspond to an indicator image that identifiably indicating particular information about an operator.

It is possible to cause the user of the information terminal apparatus 60 to know that a work machine Qi which is included in a specified area image and for which the indicator image M2i is displayed is in the remote operation status, and a work machine Qi for which the indicator image M2i is not displayed is in the actual-machine operation status or being automatically operated. Further, it is possible to cause the user of the information terminal apparatus 60 to know a communication status (example: whether communication has been established or not) between a work machine Qi and the remote operation apparatus 20 according to a display aspect of the indicator image M2i (see FIG. 6A). It is possible to cause the user of the information terminal apparatus 60 to know whether there is an abnormality in a work machine Qi or not, by the indicator image M3i (see FIG. 6B).

It is possible to, through indicator images M1i associated with the positions of work machines Qi (in FIGS. 5, i=1 to 5, 7, and 8), cause the user of the information terminal apparatus 60 to know that the current operation statuses of the work machines Qi are the remote operation status or the actual-machine operation status as the “first operation status.” On the other hand, it is possible to, through an indicator image not being associated with the position of a work machine Qi (in FIG. 5, i=6), cause the user of the information terminal apparatus 60 to know that the current operation status of the work machine Qi is the no operation status as the “second operation status”. In this meaning, the indicator image M1i corresponds to an indicator image that identifiably indicates presence/absence of an operator as the current operation status.

If the second specification operation is performed through the input interface 610 of the information terminal apparatus 60, actual-machine-related information about a work machine Qi the position of which is associated with a corresponding indicator image Mi is outputted to the output interface 620 of the information terminal apparatus 60 (see FIG. 4/STEP616→STEP618→ . . . →STEP118→STEP120→ . . . →STEP620, and FIG. 7). Thereby, it is possible to cause the user of the information terminal apparatus 60 to know through the actual-machine-related information.

(Other Forms of the First Embodiment of the Present Invention)

In the above embodiment, the image display system 10, and the first functional element 121 and the second functional element 122 constituting the image display system 10 are configured with a computer that exists separately from the remote operation apparatus 20 and the work machine 40. As another form, an image display system may be mounted on the remote operation apparatus 20 and/or the work machine 40, and the first functional element 121 and/or the second functional element 122 may be configured with the remote control device 200 and/or the actual-machine control device 400.

The information terminal apparatus 60 may be configured with the remote operation apparatus 20 or the work machine 40. When the information terminal apparatus 60 is configured with the remote operation apparatus 20, a specified area image and indicator images such as shown in FIG. 5 are outputted to the remote image output device 221.

Weather information WR may be acquired from a weather database by the first functional element 121; the weather information WR may be transmitted to the information terminal apparatus 60 by the second functional element 122; and the weather information WR may be displayed on a remote image output device constituting the output interface 620 by the control device 600 of the information terminal apparatus 60. Thereby, as shown in FIG. 8, the weather in the area where the work machines 40 exist in the specified area image or weather information about the time series of the weather is outputted to the output interface 620 of the information terminal apparatus 60. It is possible to cause the user of the information terminal apparatus 60 to know the weather in the area where the work machines 40 exist or the time series of the weather (past weather and/or predicted future weather) through the weather information.

A message MSG (notification information) inputted through the remote input interface 210 of the remote operation apparatus 20, the actual-machine input interface 41 of the work machine 40, and/or the input interface of the information terminal apparatus 60 may be outputted to the remote output interface 220, the actual-machine output interface 42, and/or the output interface 620. For example, as shown in FIG. 9, in the specified area image outputted on the output interface 620, other users and/or operators can be notified of a message MSG sent from one user and/or operator.

Second Embodiment

The following description is description about a second embodiment of the present invention.

(Configuration of Image Display Composite System)

An image display composite system as the second embodiment of the present invention shown in FIG. 10 is configured with an image display system 10, a remote operation apparatus 20 and/or a plurality of work machines 40 which are targets of remote operation by the remote operation apparatus 20, and an information terminal apparatus 60. The image display system 10, the remote operation apparatus 20, and each of the plurality of work machines 40 are configured to be capable of mutually performing communication via a common network or separate networks. Though the image display system 10 is shown as being capable of connecting to the plurality of work machines 40 in FIG. 10, the image display system 10 only has to be capable of connecting to one or more work machines 40, and the number of work machines 40 to be connected is not limited.

The concept that components (hardware) of the present invention “acquire” various kinds of information is a concept that encompasses all kinds of arithmetic processing for preparing the various kinds of information in a form that can be used in subsequent arithmetic processing, such as receiving the information, reading or retrieving the information from an internal storage device (for example, a memory) and/or an external storage device (for example, an external database server), performing, for the received, read, or retrieved information, calculation, estimation, prediction, identification, and the like by executing arithmetic processing.

(Configuration of Image Display System)

The image display system 10 is configured with a computer existing separately from the remote operation apparatus 20 and the work machine 40. The image display system 10 comprises a first functional element 121, a second functional element 122, and a database 102.

Each of the first functional element 121 and the second functional element 122 is configured with an arithmetic processing unit (a single-core processor, or a multi-core processor or a processor core constituting the multi-core processor), and reads necessary data and software from a storage device such as a memory and executes arithmetic processing described later according to the software, for the data.

The database 102 may be configured with a database server capable of mutually communicating with the image display system 10. With the database 102, operator identifiers as pieces of identification information for identifying operators OP of the work machines 40, pieces of particular information about and/or attributes of the operators OP are registered, being mutually associated. The particular information about each operator OP includes, for example, the face picture image, face illustration image, and/or name of the operator. The attributes of the operator include, for example, an organization that the operator belongs to, a language used by the operator, and/or a work job history of the operator.

The operator identifier may be information about an operator OP who is directly on board a work machine 40 or may be information about an operator OP who remotely operates work machines 40 by operating the remote operation apparatus 20. Further, when a work machine 40 works by automatic operation, the operator identifier may be information about a person who instructed the work machine 40 to perform automatic operation. As the operator identifier, for example, an ID number or the like such as an employee number is exemplified.

The particular information about and/or attributes of each operator who is remotely operating a work machine 40 through the remote operation apparatus 20 are transmitted from the remote operation apparatus 20 or the work machine 40 to the image display system 10 and then registered with the database 102, being mutually associated. The particular information about and/or attributes of the operator, and the operator identifier may be inputted by the operator through a touch panel constituting a remote input interface 210 or may be read from an electromagnetic storage medium such as a card or mobile information terminal apparatus owned by the operator by a contactless reader constituting the remote input interface 210.

With the database 102, an actual-machine identifier for identifying each work machines 40 and related information about the work machines 40 are registered in association with each other. The related information about the work machine 40 includes, in addition to static actual-machine-related information, such as the class and specifications of the work machine 40, the type of an attachment 445, and the type of crawlers constituting a lower traveling body 410, dynamic actual-machine-related information, such as a place where the remote operation apparatus 20 exists if the work machine 40 is remotely operated, content of operations performed to the work machine 40, cumulative operation time, a remaining amount of fuel, a remaining amount of soot of an exhaust gas purification device, or a remaining amount of urea water per unit period such as one day or one month.

The actual-machine identifier and the static actual-machine-related information are transmitted from each work machine 40 to the image display system 10 and then registered with the database 102 in association with each other. The actual-machine identifier and the dynamic actual-machine-related information are transmitted from the work machine 40 to the image display system 10 and then cumulatively or sequentially registered with the database 102 in association with each other. The dynamic actual-machine-related information is inputted to the actual-machine control device 400 mounted on the work machine 40 and is measured using appropriate sensors constituting an actual-machine sensor group (not shown). The dynamic actual-machine-related information includes content of operations to the work machine 40, for example, the number of adsorptions of a lifting magnet per unit time and the amount of earth and sand excavated per unit time. Further, the dynamic actual-machine-related information may include internal state variables and/or external state variables of the work machine 40. The “external state variables” are variables indicating an environment of the work machine 40, which influence the internal state variables of the work machine 40. For example, an ambient temperature around the work machine 40 is an external state variables. The “internal state variables” are variables indicating states of components of the work machine 40, for example, a water temperature in piping for engine cooling water, a temperature of hydraulic piping, a tank, and the like of a hydraulic system for causing a work mechanism 440 and the like to operate.

The particular information about and/or attributes of each operator who is actually operating a work machine 40 may be transmitted to the image display system 10 and then registered with the database 102 in association with an actual-machine identifier. The particular information about and/or attributes of the operator may be inputted by the operator through a touch panel constituting an actual-machine input interface 41 or may be read from the electromagnetic storage medium such as a card or a mobile terminal, owned by the operator by a contactless reader constituting the actual-machine input interface 41.

Further, the database 102 stores pieces of operation history information about work machines 40 in association with operator identifiers. Each piece of operation history information is stored, for example, in association with types of work machines 40 operated by an operator OP, operation time for each of the types of the work machines 40, the number of operations to be noticed, which were measured during operation of the work machines 40, work efficiencies of the operations of the work machines 40, dates and time when the operations of the work machines 40 were performed. Further, the operation history information may be stored in association with each of work sites where the work machines 40 were operated, that is, pieces of position information.

The operation history information may be information about each work machine 40 accompanying an operation of the operator OP. As such operation history information, an average value of fuel consumption of each work machine 40 (eco-operation evaluation), numerical values required for maintenance corresponding to working time of each work machine 40, and the like are exemplified.

The above pieces of operation history information is transmitted to the image display system 10 from each remote operation apparatus 20 or each work machine 40 and then registered with the database 102, being mutually associated.

In the database 102, pieces of image pickup data picked up by the actual-machine image-pickup devices 412 of each of the plurality of the work machines 40 are registered in association with the operator identifiers, respectively. The pieces of image pickup data are stored in association with the operator identifiers, respectively. Further, it is favorable that the pieces of image pickup data are registered in association with the pieces of operation history information.

(Configuration of Remote Operation Apparatus)

The remote operation apparatus 20 comprises a remote control device 200, the remote input interface 210, and a remote output interface 220.

The remote control device 200 is configured with an arithmetic processing unit (a single-core processor, or a multi-core processor or a processor core constituting the multi-core processor), and the remote control device 200 reads necessary data and software from a storage device such as a memory and executes arithmetic processing according to the software, for the data.

The remote input interface 210 comprises a remote operation mechanism 211.

The remote operation mechanism 211 includes a travel operation device, a swing operation device, a boom operation device, an arm operation device, a bucket operation device, and a hydraulic lock lever (a shutoff lever) for switching between a reception status of receiving an operation by an operator OP, who is an operating person, and a non-reception status of not receiving an operation. Each of the operation devices has an operation lever to receive a rotation operation. The operation lever of the travel operation device (a travel lever) is operated to move the lower traveling body 410 of each work machine 40, which comprises a pair of left and right crawlers. The travel lever may serve as a travel pedal. For example, a travel pedal fixed to a base part or lower end part of the travel lever may be provided. The operation lever of the travel operation device (a swing lever) is operated to move a hydraulic swing motor constituting a swing mechanism 430 of each work machine 40. The operation lever of the boom operation device (a boom lever) is operated to move a boom cylinder 442 of each work machine 40. The operation lever of the arm operation device (an arm lever) is operated to move an arm cylinder 444 of each work machine 40. The operation lever of the bucket operation device (a bucket lever) is operated to move a bucket cylinder 446 of each work machine 40. As for the shutoff lever, in the reception status, a signal for operating any work machine 40 is outputted according to an operation performed to the shutoff lever, and the work machine 40 is operated according to the signal; and, in the non-reception status, the signal for operating the work machine 40 is not outputted even if an operation is performed to the shutoff lever, and the work machine 40 is not operated.

Each of the operation levers constituting the remote operation mechanism 211 are arranged, for example, around a seat St for an operator OP to sit on as shown in FIG. 11. Though the seat St is in a form like a high-back chair with arm rests, it may be a seating portion in any form where an operator OP can be seated, such as a form of a low-back chair without a head rest or a chair without a backrest.

In front of the seat St, a pair of left and right travel levers 2110 corresponding to left and right crawlers are arranged side by side left and right. One operation lever may serve as a plurality of operation levers. For example, a left-side operation lever 2111 provided at the front of a left-side frame of the seat St shown in FIG. 11 may function as the arm lever when being operated forward and backward and function as the swing lever when being operated in left and right directions. Similarly, a right-side operation lever 2112 provided at the front of a right-side frame of the seat St shown in FIG. 11 may function as the boom lever when being operated forward and backward and function as the bucket lever when being operated in left and right directions. Lever patterns may be freely changed by operation instructions by an operator OP.

The remote output interface 220 comprises a remote image output device 221 and a remote wireless communication equipment 222 for performing communication via a network.

For example, as shown in FIG. 11, the remote image output device 221 as a display device is configured with a central remote image output device 2210, a left-side remote image output device 2211, and a right-side remote image output device 2212 that are arranged in front of, diagonally forward to the left of, and diagonally forward to the right of the seat St, each of which having an approximately rectangular screen, and a lower-side remote image output device 2213 that is arranged below the central remote image output device 2210 and has an approximately rectangular screen. The screens (image display areas) of the central remote image output device 2210, the left-side remote image output device 2211, the right-side remote image output device 2212, and the lower-side remote image output device 2213 may be the same or different in shape and size.

The screen (the image display areas) of each of the central remote image output device 2210, the left-side remote image output device 2211, the right-side remote image output device 2212, and the lower-side remote image output device 2213 may be parallel or inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left-side remote image output device 2211, the right-side remote image output device 2212, and the lower-side remote image output device 2213 may be configured with a plurality of divided remote image output devices. For example, the central remote image output device 2210 may be configured with a pair of vertically adjoining remote image output devices each of which has an approximately rectangular screen. Each of the central remote image output device 2210, the left-side remote image output device 2211, the right-side remote image output device 2212, and the lower-side remote image output device 2213 can function as input units capable of inputting information with a touch panel display or the like.

(Configuration of Work Machine)

As shown in FIG. 10, each work machine 40 comprises an actual-machine control device 400, the actual-machine input interface 41, an actual-machine output interface 42, and the work mechanism 440.

The remote control device 400 is configured with an arithmetic processing unit (a single-core processor, or a multi-core processor or a processor core constituting the multi-core processor), and the remote control device 400 reads necessary data and software from a storage device such as a memory and executes arithmetic processing according to the software, for the data.

Each work machine 40 is, for example, a hydraulic, an electric or a hybrid-drive crawler shovel (a work machine), which is hydraulic electric combined crawler shovel, and comprises the lower traveling body 410 and an upper swinging body 420 that is swingably mounted on the lower traveling body 410 via the swinging mechanism 430 as shown in FIG. 12. On a front left-side part of the upper swinging body 420, a cab 424 (an operation room) is provided. On a front central part of the upper swinging body 420, the work mechanism 440 is provided.

The actual-machine input interface 41 comprises an actual-machine operation mechanism 411 and an actual-machine image-pickup device 412. The actual-machine operation mechanism 411 comprises a plurality of actual-machine operation levers arranged similarly to those of the remote operation mechanism 211, around a seat arranged inside the cab 424.

The work machine 40 is provided with a drive mechanism or a robot that receives a signal corresponding to an operation aspect of a remote operation lever and moves an actual-machine operation lever based on the received signal is provided in the cab 424.

The actual-machine image-pickup device 412 (hereinafter also referred to as a main camera) is installed, for example, inside the cab 424. The actual-machine image-pickup device 412 picks up an image of an environment including at least a part of the work mechanism 440 through a front window and a pair of left and right side windows. For example, the actual-machine image-pickup device 412 picks up an image of an environment including at least a part of the bucket 445 which is an attachment.

Further, another actual-machine image-pickup device 412 may be appropriately added according to an aspect of implementation.

The actual-machine input interface 41 includes an actual-machine status sensor group (not shown). The actual-machine status sensor group (not shown) is configured with angle sensors for measuring a rotation angle (a raising/lowering angle) of the boom 441 relative to the upper swinging body 420, a rotation angle of the arm 443 relative to the boom 441, and a rotation angle of the bucket 445 relative to the arm 443, respectively, a swing angle sensor for measuring a swing angle of the upper swinging body 420 relative to the lower traveling body 410, an external force sensor for measuring external force acting on the bucket 445, a three-axis acceleration sensor for measuring three-axis acceleration acting on the upper swinging body 420, a position information acquisition sensor such as a GNSS (Global Navigation Satellite System), and the like.

The actual-machine output interface 42 comprises an actual-machine wireless communication equipment 422 for performing communication via a network.

The work mechanism 440 comprises a boom 441 fitted to the upper swinging body 420 in a manner that it can be raised and lowered, an arm 443 rotatably coupled with the tip of the boom 441, and the bucket 445 rotatably coupled with the tip of the arm 443. The boom cylinder 442, the arm cylinder 444 and the bucket cylinder 446, which are configured with extendable hydraulic cylinders, are fitted to the work mechanism 440. As a working portion, various attachments, such as a nibbler, a cutter, and a magnet, may be used in addition to the bucket 445.

The boom cylinder 442 intervenes between the boom 441 and the upper swinging body 420 so as to, by receiving supply of hydraulic oil, extend or contract to cause the boom 441 to rotate in a raising or lowering direction. The arm cylinder 444 intervenes between the arm 443 and the boom 441 so as to, by receiving supply of hydraulic oil, extend or contract to cause the arm 443 to rotate around a horizontal axis relative to the boom 441. The bucket cylinder 446 intervenes between the bucket 445 and the arm 443 so as to, by receiving supply of hydraulic oil, extend or contract to cause the bucket 445 to rotate around a horizontal axis relative to the arm 443.

(Configuration of Information Terminal Apparatus)

The information terminal apparatus 60 as a display apparatus is a terminal apparatus such as a smartphone, a tablet terminal, or a personal computer and comprises a control device 600, an input interface 610, and an output interface 620. The control device 600 is configured with an arithmetic processing unit (a single-core processor, or a multi-core processor or a processor core constituting the multi-core processor), and the control device 600 reads necessary data and software from a storage device such as a memory and executes arithmetic processing according to the software, for the data.

The input interface 610 is configured, for example, with a touch-panel buttons, switches, and the like. The remote output interface 620 comprises a remote image output device and wireless communication equipment for performing communication via a network.

FIG. 13 is a flowchart explaining an image display processing method of the image display system implemented by cooperation among the image display system 10, the remote operation apparatus 20, and each work machine 40 that have been described above.

The image display system starts the image display process, being triggered by activation of the remote operation apparatus 20. For example, the remote operation apparatus 20 is activated by an operator OP who is an operating person, the remote operation apparatus 20 transmits an operation signal to the image display system 10 (STEP201). The operation signal may be transmitted in response to processing after activation of the remote operation apparatus 20, such as at a timing when a work machine 40 connected to (in cooperation with) the remote operation apparatus 20 is selected by the operator OP and at a timing when the shutoff lever is switched from an ON status to an OFF status by the operator OP.

When receiving the operation signal by the first functional element 121, the image display system 10 requests the remote operation apparatus 20 to transmit the operator identifier as the identification information about the operator OP by the second functional element 122 (STEP101).

When receiving the operator identification transmission request, the remote operation apparatus 20 transmits the operator identifier to the image display system 10 (STEP202).

In the processing of STEP202, the remote operation apparatus 20 transmits the operator identifier to the image display system 10, for example, based on a result of authentication by existing user identification means such as face identification.

Further, acquisition of the operator identifier is not limited to the above aspect. For example, the operator identifier may be acquired by reading an IC card holding the operator identifier, which is identification information, or may be acquired by inputting an ID number such as an employee number inputted by a keyboard not shown. The identification information may be acquired by the operator OP performing an operation or may be acquired by a third person, such as an administrator who manages the operator OP, performing an operation.

The image display system 10 receives the operator identifier by the first functional element 121. The second functional element 122 requests the work machine 40 connected to the remote operation apparatus 20 to transmit operation information (STEP102). The operation information includes information that can be registered with the database 102, such as the dynamic actual-machine-related information.

When receiving the operation information request, the work machine 40 sequentially transmits pieces of operation information to the image display system 10 (STEP401). It is better for the work machine 40 to, at the time of transmitting the pieces of operation information, also transmit position information showing a current position of the work machine 40 obtained by the position information acquisition sensor or the like. Further, for example, when connection with the remote operation apparatus 20 is released, when the shutoff lever of the work machine 40 is ON, or the like, the work machine 40 may stop the transmission processing of STEP401.

The first functional element 121 of the image display system 10 registers the acquired pieces of operation information with the database 102 in association with the operator identifier. The second functional element 122 aggregates the pieces of operation information registered with the database 102 to generate operation history information (STEP103). In the image display system 10, a command to output the information aggregated by the second functional element 122 is generated, and the information (the operator identifier and the operation history information) is transmitted to the information terminal apparatus 60 by the second functional element 122 (STEP104). At the time of the transmission processing of STEP104, the first functional element 121 may acquire map information about the work site based on work site information inputted in advance, and the second functional element 122 may transmit the acquired map information to the information terminal apparatus 60.

When receiving the operator identifier and the operation history information, the information terminal apparatus 60 displays the identification information and the operation history information on the remote image output device 221 of the output interface 620 as specification information (STEP601).

If a work machine 40 to be connected to the remote operation apparatus 20 is switched from the one work machine to another work machine, the image display process shown in FIG. 13 may be started from when the work machine 40 that has been switched to and is connected to the remote operation apparatus 20 is requested to transmit operation information (STEP102). That is, since the operator identifier has been already acquired by the image display system 10, the process until requesting the operator identifier (STEP201→STEP101→STEP202) can be omitted. Further, it is better for the work machine 40 the connection of which with the remote operation apparatus 20 has been released to stop transmission of operation information.

FIG. 14 shows a display aspect on the output interface 620 of the information terminal apparatus 60 as a display apparatus. As shown in FIG. 14, together with a map of a certain work site AR (hereinafter also referred to as a yard), icons A to F of six work machines 40 are displayed at positions on the map where the work machines 40 are arranged, respectively, on the center of the screen of the output interface 620.

Near the icons of the work machines 40, icons X1 to X6 of operators OP who operate the work machines 40, respectively, are displayed. As the icons X1 to X6 of the operators OP, pictures of the operators OP registered in advance may be displayed, or avatars indicating the operators OP may be displayed. Each of the icons only has to be such that can identify one operator among a plurality of operators, without being limited to an image, and may be, for example, character information indicating a name.

On the upper left of the output interface 620 of the information terminal apparatus 60, an icon X7 indicating a user of the information terminal apparatus 60 is displayed. Below the icon X7 on the output interface 620, various display menu choices may be arranged.

FIG. 15 shows a display in a case where the icon A of the work machine 40 shown in FIG. 14 is selected. The selection can be made, for example, by performing an operation such as tapping of a keyboard and/or a menu choice.

As shown in FIG. 15, when any of the icons A to F of the work machines 40 is selected, working information showing a working status of the work machine 40 and particular information about the operator OP operating the work machine 40 are displayed.

As the working information, for example, the model number of the work machine 40, video at the time of the work machine 40 being operated, the remaining fuel of the work machine 40, working time after the last maintenance of the work machine 40, time at the work site, and a map of the work site are exemplified. The working information may be any information about the work machine 40, without being especially limited, and, for example, information required for maintenance of the work machine 40 and the like are exemplified.

As the particular information about the operator OP, the name of the operator OP who is on board the work machine 40 and the icon of the operator OP are displayed in FIG. 15. It is better to display the identification information, for example, by color so that it can be identified which of a status of the operator OP operating the work machine 40, being directly on board the work machine 40, a status of the operator OP operating the work machine 40 via the remote operation apparatus 20, and a status of the work machine 40 having been given an automatic operation instruction is the current status.

Information displayed on the output interface 620 of the information terminal apparatus 60 is not limited to the above information. A call button for starting a call using a voice call function of another terminal, for example, a communication apparatus (a smartphone) carried by an administrator or an operator and/or the work machine 40, the remote operation apparatus 20, or the information terminal apparatus 60, and an icon indicating a remaining amount of urea water may be displayed.

For example, when the icon X1 of an operator OP is selected in FIG. 15, a display shown in FIG. 16 is displayed. In FIG. 16, cumulative time spent for performing operation of “lifting magnet”, “breaker”, “high reach crane”, and/or “nibbler” by each of operators 1 to 5 is displayed. The start date or start time, and the end date or end time of aggregating the cumulative time can be freely determined. The operators 1 to 5 show any five operators among the icons X1 to X6 of the operators displayed in FIG. 14. That is, in FIG. 16, information associated with the operators 1 to 5 who operate work machines 40 in the certain work site AR is displayed.

Further, for example, by the user making a selection from a menu bar not shown, the operators may be displayed in descending order of cumulative time or in ascending order of cumulative time. In FIG. 16, the operators are displayed in descending order of cumulative time from the left side of the screen.

Furthermore, the operation histories of the operators may be displayed being arranged based on the cumulative number of operations instead of the cumulative time. As the cumulative number of operations, for example, the number of days when attachments such as the “lifting magnet”, “breaker”, “high reach crane”, and “nibbler” are operated is exemplified.

By such a display being displayed, it becomes possible to visually display the operators' operation experiences. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when a scroll-down operation is performed in FIG. 16, a display shown in FIG. 17 is displayed. In FIG. 17, the number of times each of the “lifting magnet”, “breaker”, “high reach crane”, and “nibbler” operations was performed in an organization to which the operators 1 to 5 belong during a period from 2019 to 2021 is displayed. By such a display being displayed, it becomes possible to grasp the frequency of particular work performed in business.

For example, when the icon X1 of the operator OP is selected in FIG. 15, a display shown in FIG. 16 is displayed. In FIG. 16, total time of operation of a work machine/work machines 40 by each of the operators 1 to 5 in a time zone of a certain date. By such a display being displayed, it becomes possible to grasp a ratio of content of work performed by each operator. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when the icon X1 of the operator OP is selected in FIG. 15, a display shown in FIG. 19 is displayed. In FIG. 19, work efficiency of the work machine 40 for the operator 1 in time zones of a certain date is shown. For example, in the case of operating the “lifting magnet”, the work efficiency is a value of operation time (minutes)/the number of adsorptions. By such a display being displayed, it is possible to grasp, for example, the ability of the operator OP. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when the icon X1 of the operator OP is selected in FIG. 15, a display shown in FIG. 20 is displayed. In FIG. 20, an average of fuel consumption per unit time (l/Hr) in a case where the operator 1 has operated a plurality of work machines 40 is shown. In FIG. 20, for example, it can be read that, for one work machine (a black triangle (▴)), the amount of fuel consumed for one hour from 12:00 to 13:00 is 1.2 liters. By such a display being displayed, it is possible to grasp the operator's operations with little loss, that is, work efficiency. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when the icon X1 of the operator OP is selected in FIG. 15, a display shown in FIG. 21 is displayed. In FIG. 21, total operation time of operation of work machines 40 of the operators 1 to 4 in a certain work site is shown for each time zone. By such a display being displayed, it is possible to grasp work time of each of the operators 1 to 4 in each time zone. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when the icon X1 of the operator OP is selected in FIG. 15, a display shown in FIG. 22 is displayed. In FIG. 22, work efficiency of the work machine 40 for each of the operators 1 and 2 is shown. Since calculation of the work efficiency is the same as described above, description thereof will be omitted. By such a display being displayed, it is possible to grasp the work efficiency of each of the operators 1 and 2 in each time zone. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when the icon X1 of the operator OP is selected in FIG. 15, a display shown in FIG. 23 is displayed. In FIG. 23, the number of times of work of the work machine 40 to be noticed, for each of the operators 1 and 2 is shown. By such a display being displayed, it is possible to grasp the number of times of the work to be noticed, for each of the operators 1 and 2 in each time zone. As for the work to be noticed, for example, if the operation history information includes operations corresponding to sudden operations (start, swing, or attachment operations) and a sudden stop, the number of times of such work is counted. As for the work to be noticed, not only the number of sudden operations (start, swing, and attachment operations) and sudden stops but the number of times of such work or operation that destabilizes the operation condition of the work machine 40 may be counted. Therefore, it is possible to recognize work time zones in which safety is considered. The display is not limited to being displayed by selection from the icons X1 to X6. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when an icon indicating weather (not shown) is selected in FIG. 15, a display shown in FIG. 24 is displayed. In FIG. 24, precipitation amounts and temperatures at a place where the work machine 40 operated by the operator OP is located are shown. By such a display being displayed, it is possible to grasp weather at a place where the work machine 40 is/was located. Thereby, it is possible to confirm whether the quality of work depends on weather or not. In the display shown in FIG. 24, wind speeds may be displayed instead of temperatures as shown in FIG. 25. The display is not limited to being displayed by selection of an icon. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

For example, when an icon indicating video is selected in FIG. 15, a display shown in FIG. 26 is displayed. In FIG. 26, for each of the operators OP, the type of an operated work machine 40 (a construction machine name in FIG. 26), a recording start date, recording start time, the type of a camera used for recording (Main), the length of time of video, a link to the uploaded video, and a deletion status of video data are shown. By such a display being displayed, it is possible to, for each operator OP, confirm an aspect of operation of the work machine 40 by video taken by the actual-machine image-pickup device 412. For example, by identifying time during which an operation with little loss was performed, based on the fuel consumption information shown in FIG. 20 and confirming video corresponding to the time, the user can confirm the aspect of the operation with little loss. The display is not limited to being displayed by selection of an icon. A menu bar may be appropriately displayed so that the user can make a selection from the menu bar to display the display.

Based on the above example, it is possible to, even if an operator OP transfers from one work machine 40 to another work machine 40, acquire appropriate work results of the operator OP.

Further, for example, it is possible to facilitate calculation of remuneration paid to operators OP. Specifically, since aggregation of work for each operator OP is not conventionally performed, it is difficult to, when an operator OP transfers among a plurality of work machines 40 to perform work, calculate remuneration based on work results. Especially, in the case of operating a work machine 40 using the remote operation apparatus 20, an operator OP can perform an operation without going to the work machine 40. Therefore, it is expected that the operator OP operates more work machines 40 in comparison with the case of actually getting on board work machines 40, and, therefore, it is necessary to manage operation information about each work machine 40 for each operator OP. Further, since it is expected that a work machine 40 is operated by a plurality of operators OP in turn, it is necessary to differentiate operation information among operators OP. Since work results of each operator OP can be acquired even in such a situation, it is possible to easily calculate remuneration of each operator OP based on work results. Furthermore, it becomes possible to, by adding up work results of operators OP, calculate work results of each group (party or company) to which the plurality of operators OP belong, and it is also possible to confirm the rank of particular operators OP in the group.

The description of the above example has been made on the assumption that the display apparatus is the information terminal apparatus 60. The display apparatus is not limited to such an aspect. For example, the remote image output device 221 of the remote operation apparatus 20 may be implemented as the display apparatus. Further, the number of information terminal apparatuses 60 is not limited to one. The display apparatus may be implemented with a plurality of information terminal apparatuses 60. For example, the information terminal apparatus 60 may be installed near each of the cab 424 of the work machine 40 and the remote operation apparatus 20.

Further, in the description of FIG. 16 and the like, the operators 1 to 5 are described as operators who operate the work machines 40 in the work site AR. As for the operators 1 to 5, however, operators who work in work sites different from the work site AR may be freely selected and displayed. Further, the number of operators to be displayed in FIG. 16 and the like, and the types of attachments shown in FIG. 17 and the like may be appropriately changed.

Furthermore, in the description of the above example, it is described that the image display process is executed by the image display system 10. The image display process, however, may be performed by the remote operation apparatus 20 or the work machine 40.

Further, in the description of the above example, a request for an operator identifier by the image display system 10 is made only once after the remote operation apparatus 20 starts. There may be, however, a case where, after the remote operation apparatus 20 starts, an operator OP operating the remote operation apparatus 20 transfers operation to another operator OP, and the other operator OP operates the remote operation apparatus 20 midway. In such a case, it is necessary for the image display system 10 to acquire the operator identifier of each operator OP. In order to realize this, the image display system 10 may request transmission of an operator identifier, for example, when seating on the seat St is detected again after detection of an operator OP leaving from the seat St. As for detection of an operator OP leaving from the seat St, the detection is made in a case where the face of an operator OP is not detected on a picked-up image of a camera used for face authentication, a case where a load is not detected by a load sensor provided on the seat St, or the like.

Further, replacement of an operator OP may be detected by change of an operator identifier. For example, replacement of an operator OP may be detected by using an authentication result by user authentication means so as to confirm whether an operator identifier has not been changed. Specifically, the image display system 10 may acquire an authentication result of authentication by user authentication means such as face authentication at regular intervals and confirm whether an operator identifier has not been changed, based on the authentication result. Thereby, it becomes possible to certainly set an operator identifier.

It becomes possible to, even when an operator OP performing an actual-machine operation of a work machine 40 transfers the operation, and another operator OP performs the operation midway, certainly set an operator identifier by similarly detecting the replacement of the operator OP.

In the description of the above example, the function of arranging the operators 1 to 5 in descending or ascending order of cumulative time or the cumulative number of operations is described in FIG. 16. The operators, however, may be arranged and displayed in descending order of operation score. The operation score is an indicator (a numerical value) set such that the higher the evaluation of an operator OP in the operation history information is, the higher the score is. As conditions for a high operation score, for example, the cumulative time and cumulative operation time described above being long and the operation experience being sufficient, the work efficiency being high, the average fuel consumption being low, and the amount of work to be noticed being small are exemplified. Further, at the time of calculating an operation score, it may be calculated by setting evaluation of operation history information at the time of bad weather higher. Further, the work efficiency may be evaluated based on whether the amount of earth and sand excavated per unit time is large or not, whether a deviation of an operation amount relative to a standard operation amount is small or not, and the like. The operation score may be calculated by compositely evaluating a plurality of pieces of operation history information.

In the image display system of the present invention, it is preferable that, for each of work machines, if the current operation status of the work machine is a first operation status, the image display system causes an output interface of an information terminal apparatus to output a corresponding indicator image among the indicator images in association with the position of the work machine, and, if the current operation status of the work machine is a second operation status, the image display system does not cause the output interface of the information terminal apparatus to output the corresponding indicator image in association with the position of the work machine.

According to the image display system with the above configuration, it is possible to, on the output interface of the information terminal apparatus, cause the user of the information terminal apparatus to know that the current operation status of a work machine is in the first operation status through a corresponding indicator image associated with the position of the work machine existing in a specified area. On the other hand, it is possible to, on the output interface of the information terminal apparatus, cause the user of the information terminal apparatus to know that the current operation status of the work machine is in the second operation through the fact that an indicator image is not associated with the position of the work machine existing in the specified area.

In the image display system of the present invention, it is preferable that, for each of the work machines, the image display system causes the output interface of the information terminal apparatus to output the corresponding indicator image identifiably indicating at least one of the following, that is, presence/absence of an operator, particular information about the operator, and attributes of the operator, as the current operation status of the work machine in association with the position of the work machine.

According to the image display system with the above configuration, it is possible to, on the output interface of the information terminal apparatus, cause the user of the information terminal apparatus to know presence/absence of an operator, particular information about the operator, and/or attributes of the operator as the current operation status, through the corresponding indicator image associated with the position of the work machine existing in a specified area in a specified area image.

In the image display system of the present invention, it is preferable that, for each of the work machines, the image display system causes the output interface of the information terminal apparatus to output the corresponding indicator image identifiably indicating a status of communication with a remote operation apparatus as the current operation status of the work machine, in association with the position of the work machine.

According to the image display system with the above configuration, it is possible to, on the output interface of the information terminal apparatus, cause the user of the information terminal apparatus to know a communication state (example: whether communication has been established or not) between the work machine and the remote operation apparatus as the current operation status, through the indicator image associated with the position of the work machine existing in the specified area in the specified area image.

In the image display system of the present invention, it is preferable that, for each of the work machines, the image display system causes the output interface of the information terminal apparatus to output, in addition to the specified area image, weather of the area where the work machine exists or weather information about a time series of the weather.

According to the image display system with the above configuration, it is possible to cause the user of the information terminal apparatus to know the weather in the area where the work machine exists or the time series of the weather (predicted weather of and after the current time and time zone) through the weather information outputted to the output interface of the information terminal apparatus.

In the image display system of the present invention, it is preferable that, if a specification operation is performed through an input interface of the information terminal apparatus, the image display system causes the output interface of the information terminal apparatus to output actual-machine-related information related to any of the work machines, the position of the work machine being associated with a specified indicator image among the indicator images.

According to the image display system with the above configuration, it is possible to, if a specification operation is performed through the input interface of the information terminal apparatus, cause the user of the information terminal apparatus to know through the actual-machine-related information about a corresponding work machine outputted to the output interface.

In the image display system of the present invention, it is preferable that, for each of the pieces of identification information, the image display system causes the output interface of the information terminal apparatus to output pieces of operation history information about one or more of the work machines corresponding to a piece of operation history information about a corresponding operator.

According to the image display system with the above configuration, it is possible to cause the user of the information terminal apparatus to know the operation history information associated with each of the plurality of operators.

In the image display system of the present invention, it is preferable that, if a predetermined operation for any of the work machines is performed in a state in which the pieces of operation history information are displayed on the output interface of the information terminal apparatus, the image display system causes the output interface of the information terminal apparatus to output map information.

According to the image display system with the above configuration, it is possible to, when a predetermined operation is performed for any work machine, cause the user of the information terminal apparatus to know the map information. Further, by displaying a display on the information terminal apparatus, it is possible to, if the user of the information terminal apparatus is an operator, display a display corresponding to an operation by the operator.

In the image display system of the present invention, it is preferable that the image display system causes the output interface of the information terminal apparatus to output the plurality of pieces of operation history information in order of numerical values included in the plurality of pieces of operation history information.

According to the image display system with the above configuration, it is possible to cause the user of the information terminal apparatus to know the pieces of operation history information in order of numerical values included in the pieces of operation history information. If the user of the information terminal apparatus is an administrator, the administrator can compare, for example, the plurality of operators according to lengths (sizes) of operation histories.

In the image display system of the present invention, it is preferable that the image display system causes the output interface of the information terminal apparatus to output the pieces of operation history information for each work area determined in advance.

According to the image display system with the above configuration, pieces of operation history information corresponding to work areas are ranked for each operator. Therefore, it is possible to cause the user of the information terminal apparatus to know work efficiencies corresponding work areas (work environments) for each operator.

In the image display system of the present invention, it is preferable that, the work machines are remotely operated.

According to the image display system with the above configuration, it is possible to display operation history information about each operator on the display apparatus for each of remotely operated work machines.

In the image display system of the present invention, it is preferable that the image display system comprises a remote operation apparatus for remotely operating the work machines, and the remote operation apparatus switches an operation target from one work machine to another work machine.

According to the image display system with the above configuration, it becomes possible to display pieces of operation history information about a plurality of work machines operated by one remote operation apparatus, in association with operators, respectively. For example, there may be a case where two operators operate one work machine by switching a remote operation apparatus connected to the one work machine. Even in such a case, it is possible to display operation history information about each of the operators on the display apparatus.

REFERENCE SIGNS LIST

- 10 image display system, 20 remote operation apparatus, 40 work machine, 41 actual-machine input interface, 42 actual-machine output interface, 102 database, 121 first functional element, 122 second functional element, 200 remote control device, 210 remote input interface, 211 remote operation mechanism, 220 remote output interface, 221 remote image output device, 400 actual-machine control device, 424 cab (operation room), 440 work mechanism, 445 attachment (bucket)

Number	Date	Country	Kind
2021-197010	Dec 2021	JP	national
2021-197266	Dec 2021	JP	national

IMAGE DISPLAY SYSTEM, IMAGE DISPLAY COMPOSITE SYSTEM, AND IMAGE DISPLAY METHOD

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (2)

PCT Information