The disclosure generally relates to controlling machines and, more particularly, to controlling and operating machines remotely at least in part with a portable computing device.
Machines including excavators, loaders, dozers, motor graders, haul trucks, and other types of equipment are used to perform a variety of tasks. During the performance of these tasks, the machines may operate in situations that are unsuitable or undesirable for a human operator such as hazardous conditions, extreme environmental conditions, uncomfortable for an operator, or at work locations remote from civilization. Because of these factors, the completion of some tasks by an onboard operator may be dangerous, expensive, and otherwise undesirable.
One solution is to remotely control and operate the machines. However, operation of the machines often requires skilled operators. Still further, additional skills are required to operate the machines remotely. As a result, skilled machine operators that also have the ability to operate machines remotely may be difficult to locate. The lack of skilled operators may lead to delays in projects, lost income, and increased costs. Additional training may be required to develop operators qualified to operate machines remotely. The additional training may increase the costs of operation and cause additional delays.
U.S. Pat. No. 6,739,078 discloses a system for remotely controlling equipment such as a backhoe via a data network. A user may provide movement instructions via a graphical user interface at a user PC to a programmable controller interfaced to the data network and the hydraulic movement systems of the equipment. The graphical user interface includes a visual representation portion and a user control portion. A visual representation of the moveable elements of the equipment is provided to the user via the graphical user interface, and the user inputs movement instructions via the user control portion of the graphical user interface. The visual representation may be generated in response to movements of the components of the equipment.
The foregoing background discussion is intended solely to aid the reader. It is not intended to limit the innovations described herein, nor to limit or expand the prior art discussed. Thus, the foregoing discussion should not be taken to indicate that any particular element of a prior system is unsuitable for use with the innovations described herein, nor is it intended to indicate that any element is essential in implementing the innovations described herein. The implementations and application of the innovations described herein are defined by the appended claims.
In one aspect, a system for remotely operating a machine includes a portable computing device, wherein the portable computing device has a portable device controller and is removably mountable at a machine and at a remote control console. A machine includes a plurality of sensors for generating signals indicative of operating conditions of the machine, a visual image sensor for generating visual image signals indicative of a point of view relative to the machine, and a machine mounting location for removably mounting the portable computing device thereat. A machine instrument array includes, upon operatively mounting the portable computing device at the machine mounting location, a machine display device and a machine input device. The machine display device is configured to display information related to operation of the machine, and the machine input device is configured to receive an input command from an operator. A remote control console is configured to control the machine remotely and includes a console mounting location for removably mounting the portable computing device thereat and a console instrument array that includes, upon operatively mounting the portable computing device at the console mounting location, a console display device and a console input device. The console display device is configured to display information related to operation of the machine and the console input device is configured to receive an input command from an operator. A point of view display is configured to display an image of the point of view relative to the machine. A controller includes the portable device controller. Upon operatively mounting the portable computing device at the machine, the controller is configured to receive signals from the plurality of sensors of the machine, determine the operating conditions of the machine based upon the signals, and transmit a plurality of machine signals to the machine instrument array indicative of the operating conditions of the machine. For each operating condition, the machine instrument array has a predetermined configuration. Upon operatively mounting the portable computing device at the remote control console, the controller is further configured to receive signals from the plurality of sensors of the machine, determine the operating conditions of the machine based upon the signals, and transmit visual image signals indicative of the point of view relative to the machine to the remote control console. The controller is further configured to display point of view images on the point of view display based upon the visual image signals, transmit signals from the remote control console to the machine to direct the machine to perform desired operations, and transmit a plurality of console signals to the console instrument array indicative of the operating conditions of the machine. The console signals further define a configuration of the console instrument array and, for each operating condition, the configuration of the console instrument array is generally identically to the predetermined configuration of the machine instrument array.
In another aspect a controller implemented method of remotely operating a machine includes determining, for each of a plurality of operating conditions of the machine, a predetermined configuration of a machine instrument array, removably mounting a portable computing device having a portable device controller on a remote control console, and transmitting to the remote control console signals from a plurality of sensors operatively associated with the machine indicative of the operating conditions of the machine. The method further includes determining the operating conditions of the machine based upon the signals received from the plurality of sensors, determining a plurality of console signals indicative of the operating conditions of the machine, and defining a configuration of a console instrument array of the remote control console. For each of the plurality of operating conditions, the configuration of the console instrument array is generally identically to the predetermined configuration of the machine instrument array. The method still further includes transmitting the plurality of console signals to the console instrument array of the remote control console.
In still another aspect, a system for remote control operation of a machine includes a portable computing device that has a portable device controller and is removably mountable at the machine and at a remote control console. The machine includes a prime mover, a plurality of sensors for generating signals indicative of operating conditions of the machine, a display for displaying images related to the operating conditions of the machine, a visual image sensor for generating visual image signals indicative of a point of view relative to the machine, and a machine mounting location for removably mounting the portable computing device thereat. The machine further has a machine instrument array including, upon operatively mounting the portable computing device at the machine mounting location, a machine display device and a machine input device. The machine display device is configured to display information related to operation of the machine and the machine input device is configured to receive an input command from an operator. A remote control console is configured to control the machine remotely and includes a console mounting location for removably mounting the portable computing device thereat, a console instrument array, and a point of view display for displaying an image of the point of view relative to the machine. Upon operatively mounting the portable computing device at the console mounting location, the console instrument array includes a console display device and a console input device. The console display device is configured to display information related to operation of the machine and the console input device is configured to receive an input command from an operator. A controller, including the portable device controller, upon operatively mounting the portable computing device at the machine, is configured to receive signals from the plurality of sensors of the machine, determine the operating conditions of the machine based upon the signals, and transmit a plurality of machine signals to the machine instrument array indicative of the operating conditions of the machine. For each operating condition, the machine instrument array has a predetermined configuration. The controller, including the portable device controller, upon operatively mounting the portable computing device at the remote control console, is further configured to receive signals from the plurality of sensors of the machine, determine the operating conditions of the machine based upon the signals, and transmit visual image signals indicative of the point of view relative to the machine to the remote control console. The controller is still further configured to display point of view images on the point of view display based upon the visual image signals, transmit signals from the remote control console to the machine to direct the machine to perform desired operations, and transmit a plurality of console signals to the console instrument array indicative of the operating conditions of the machine. The console signals further define a configuration of the console instrument array and, for each operating condition, the configuration of the console instrument array is generally identically to the predetermined configuration of the machine instrument array.
A work implement such as a blade or moldboard 20 extends downward from the frame 11. The moldboard 20 may be mounted on a blade tilt adjustment mechanism 21 that is supported by a rotatable circle assembly 22 operatively connected to the blade tilt adjustment mechanism 21. A variety of hydraulic cylinders or actuators may be provided for controlling the position of the moldboard 20. For example, circle assembly 22 may be supported by a pair of blade lift actuators 23 (with only one visible in
The machine 10 may be equipped with a plurality of sensors or sensing devices that gather data from various components and systems and generate signals that are directly or indirectly indicative of the performance and operating conditions of the machine. The sensors may generate signals indicative of operating conditions of the machine. Sensors may be associated with, for example, the engine 12, a transmission (not shown), a torque converter (not shown), the front wheels 13, the rear wheels 14, the operator cab 15, the moldboard 20, various actuators such as the blade lift actuators 23, the center shift cylinder 24, the blade tip cylinder 25, fluid supplies (not shown), operator input devices, a parking brake and/or other systems and components of machine 10. These sensors may automatically gather real-time data such as the operation of engine 12, the position of and load on the work implement, fluid pressure, flow rate, temperature, contamination level, and/or viscosity, fluid (i.e., fuel, oil, water, etc.) consumption rates, electric current and/or voltage levels, loading levels (e.g., payload value, percent of maximum allowable payload limit, payload history, payload distribution, etc.), transmission output ratio, and other desired information.
In addition, various sensors may be associated with the machine 10 that may be used to determine machine travel characteristics (e.g., speed, acceleration, torque, slip rate, etc.) as well as the position and orientation of machine 10. For example, an accelerometer 27 may be provided on the machine 10 to provide an acceleration signal indicative of measured acceleration of the machine 10 relative to a gravity reference. In one example, the accelerometer 27 may provide measurements in six degrees of freedom (i.e., fore-aft, lateral, and vertical directions as well as pitch, roll and yaw). In some circumstances, it may be desirable to position the accelerometer 27 generally adjacent operator cab 15 so that movement sensed by the accelerometer somewhat matches movement sensed by an operator.
Still further, a pitch rate sensor 28 (e.g., a gyroscope) may be provided on the machine 10. The pitch rate sensor 28 may be used to provide a pitch rate signal indicative of a pitch rate of the machine 10. As the machine 10 moves, the pitch rate will be indicative of the rate of change of the pitch angle of the machine. The pitch rate sensor 28 may also be used to determine the pitch and roll of the machine 10.
A position sensor 29 may sense a position of the machine 10. The position sensor 29 may include a plurality of individual sensors that cooperate to provide signals to controller 51 to indicate the position of the machine 10. The controller 51 may determine the position of the machine 10 as well as its orientation (i.e., the direction machine 10 is facing). In some instances, the position sensor 29 may be used to determine the pitch and roll of the machine 10. The position sensor 29 may be a series of global positioning system sensors, an odometer or other wheel rotation-sensing sensor, a perception based system or may use other systems such as lasers to determine the position of machine 10.
As depicted schematically in
Each input device may function as a machine input device for providing commands or data input such as inputting information, changing operations, and issuing commands to the machine 10 and to a remote system 120. More specifically, machine 10 may include a plurality of fixed or stationary input devices such as buttons, knobs, dials, levers, joysticks, and other controls that are permanently mounted within the operator cab 15. One or more of the stationary input devices may have a fixed functionality (referred to herein as a fixed function input device 33) such that they are always used to control the same function (e.g., on/off switches, cab temperature controls, pedals, radio controls). One or more of the stationary input devices may have a modifiable or changeable functionality (referred to herein as a modifiable function input device 34) such that they may be modified to control the input of different functions (e.g., cab temperature, machine lights, or engagement of auto blade position set features). Examples of modifiable function input devices 34 include a touch screen display with a computer-generated image, a knob adjacent a computer display, or any other desired input device.
A control system 50 may be provided to control the operation of the machine 10. The control system 50, as shown generally by an arrow in
The controller 51 is shown in
The controller 51 may be a single controller or may include more than one controller disposed to control various functions and/or features of the machine 10. In one embodiment depicted in
As depicted in
The display 103 may be configured as a touch screen to also operate as a portable device input. The wireless interface 104 may provide a communications channel between the machine controller 52 and the portable computing device 100 as well as between the controller 51 and a remote system 120. In one embodiment, the wireless communication between the machine controller 52 and the portable computing device 100 may be via a Bluetooth® communications system or protocol. Other communications systems and protocols are contemplated. In an alternate embodiment, a wired communications interface (not shown) may be provided to effect the connection between the machine controller 52 and the portable computing device 100. Communication between the machine controller 52 and the portable computing device 100 may be achieved by a wireless connection that occurs once the portable computing device is within a predetermined distance of the machine 10. In another embodiment, communication between the machine controller 52 and the portable computing device 100 may be achieved by a wired connection upon mounting the portable computing device in or on the machine 10. In other words, portable computing device 100 may be configured to communicate with the machine controller wirelessly or through a wired connection.
Components of the portable computing device 100 may supplement or replace some of the components of machine 10. For example, the functionality of controller 51 may be distributed so that certain functions are performed by the machine controller 52 and other functions are performed by the portable computing device 100. In addition, sensors, displays and input devices of the portable computing device 100 may supplement or replace some of the sensors, displays, gauges, and input devices of machine 10. For example and as described in further detail below, the display 103 of the portable computing device 100 may be used to supplement or replace displays permanently mounted on the machine 10. Similarly, the wireless interface 104 of the portable computing device 100 may also be used to supplement or replace a wireless interface of the machine 10.
The camera 105 of the portable computing device 100 may be used to capture images of the work site or the machine 10. These images may be stored in the machine controller 52, in the portable computing device 100, or in a remote system 120 wirelessly connected to the controller 51. Further, the portable computing device 100 may be mounted within operator cab 15 so that camera 105 may be used to monitor actions of an operator such as tracking an operator's eyes to monitor for potential signs of fatigue. Microphone 106 of portable computing device 100 may be used to receive voice commands from an operator and provide the voice commands to the controller 51. The controller may thus be able to operate components or control certain aspects of the operation of machine 10 based upon those voice commands and thus reduce the need for additional input devices. The microphone 106 may also be used to communicate or record notes regarding the operations at the work site or the operation of the machine 10 such as noted hazards or required maintenance items. The global positioning sensor 107 of the portable computing device 100 may used to supplement or replace the sensors of position sensor 29, if desired. As a result, the display 103 and other components of the portable computing device 100 may supplement or combine with the machine display devices and the machine input devices to form a machine instrument array made up of the instrument array 30 and the portable computing device.
In an alternative embodiment, the portable computing device 100 may control and operate all aspects of the machine 10 and thus the portable computing device 100 may function as the controller 51. Regardless of whether the connection between the machine controller 52 and the portable computing device 100 is wired or wireless and regardless of whether controller 51 includes machine controller 52, aspects of the control system 50 may be located remote from the machine 10 and controller 51 may communicate wirelessly to remote system 120 that forms a part of the control system 50.
The portable computing device 100 may store as data codes a plurality of user or operator preferences such as the temperature of the operator cab 15, the settings of the seat within the operator cab 15, input device sensitivity, and desired display images based upon certain operating conditions. Upon docking the portable computing device 100 or moving it within a predetermined range in the case of a wireless connection, the machine controller 52 and the portable computing device may communicate so that controller 51 will operate with the user preferences saved within the portable computing device. The data codes may further include a user identification data code that is specific or unique to each machine operator. As a result, upon docking the portable computing device 100, the controller 51 may determine the identity of the machine operator based upon the user identification data code.
The controller 51 may be configured so that machine 10 is inoperable without portable computing device 100. For example, portable computing device 100 may have saved therein data codes such as authorization keys that prevent the operation of the machine controller 52 without such authorization keys. By requiring appropriate authorization keys, it may be possible to limit an operator's use to only certain types or models of machines 10. More specifically, each operator may be assigned a specific portable computing device 100 and that portable computing device may be set up so as to limit an operator to operate only certain types or models of machines. Still further, the portable computing device 100 may limit an operator to operating one or more specific machines.
The controller 51 may be further configured to limit the operation of machine 10 based upon data codes in the form of limited access keys stored within portable computing device 100. The limited access keys may be used to limit the types of operations performed with the machine 10 as well as limit the time or location at which the machine is operated. For example, a limited access key may be used to prevent an inexperienced operator from performing certain operations (e.g., to prevent operation without sufficient training) In another example, an operator may be prohibited from clearing certain types of data such as performance data as well as fault or error codes. The limited access keys may further be used to prevent a machine 10 from being moved outside of a predetermined area (e.g., outside of a global positioning system defined fence) or from being moved or performing certain operations outside of a predetermined range of hours or times.
Machine operators may be provided with one set or type of limited access keys while maintenance personnel may be provided with a different set or type of limited access keys. As a result, maintenance personnel may be prevented from performing certain operations such as moving a machine or operating certain implements and operators may be prevented from accessing certain components or panels of the machine that should only be accessed by maintenance personnel. Through such limited access keys, a work site manager or machine owner may be able to match the use and access to the machines 10 based upon the ability and authorization of the various personnel. Still further, a record of such use and access may be stored within the portable computing device 100 and/or communicated wirelessly to remote system 120.
As described above, the operator cab 15 may include a plurality of fixed or stationary display devices such as gauges 31 and displays 32 that are permanently mounted within the operator cab 15 as part of the machine instrument array. Other display devices may be removably mounted on the machine 10 such as within the operator cab 15. In one example, if the portable computing device 100 is removably positioned within the operator cab 15, the display 103 of the portable computing device may be used as an additional or portable display within the operator cab. As with the displays 32 that are fixed within the operator cab 15, the display 103 of a portable computing device 100 may be used to display any type of image including those that resemble gauges. The gauges 31, the displays 32, and the display 103 of the portable computing device 100 may interact to form a display system such as the instrument array 30 of machine 10.
As also described above, the operator cab 15 may include a plurality of fixed or stationary input devices such as fixed function input devices 33 and modifiable function input devices 34 that are permanently mounted within the operator cab 15. Other input devices may be removably mounted on the machine 10 such as within the operator cab 15. If the portable computing device 100 is removably positioned within the operator cab 15 and the display 103 includes touch screen functionality, a portion of the display may be used as an input device with modifiable functionality. Still further, the portable computing device 100 may include other components such as microphone 106 that may be used as input devices to control the operation of machine 10. The fixed function input devices 33, the modifiable function input devices 34, and aspects of the portable computing device 100 may interact to form an input system of machine 10.
The controller 51 may be configured to control the images displayed on each of the modifiable displays and gauges. The controller 51 may display different images depending on the operation being performed and the operating conditions of the machine 10. In one example, upon starting machine 10, the displays may indicate the machine status, the temperature of various fluids, and the fuel level. After reporting and confirming the initial levels, it may be desirable to only display this information upon a material change, at predetermined time intervals, or upon the request of an operator.
The controller 51 may change the images being displayed based upon the operation being performed. For example, in
When the motor grader is performing a grading operation, it may be desirable to only depict some of the images displayed in
In another operation such as the motor grader operating on a road, different images may be displayed as depicted in
If an emergency alert or alarm occurs, the controller 51 may display an image on display 32 related to the emergency alert or alarm as well as instructions as to the appropriate steps to be taken or operations to be performed. For example, if one or more of the fluid temperatures exceeds a predetermined threshold, the controller 51 may generate a warning for the operator on the display 32.
Although described above with respect to a motor grader, the present disclosure may be applicable to many other types of machines. For example, when operating a wheel loader, controller 51 may display an indication of how close the wheels are to slipping. This information may be used by an operator to maximize or increase the performance of the wheel loader. In another example, when loading an excavator or a wheel loader, the controller 51 may display mass or weight of the load in the work implement of the machine.
A visual image system 55 such as a camera system may be provided for generating visual images indicative of a point of view relative to the machine 10. The visual image system 55 may include a plurality of visual image sensors such as cameras (not shown) for generating visual image signals. The visual image signals may be transmitted wirelessly to a system remote from machine 10. In doing so, the visual image signals may be processed to some extent by controller 51 at machine 10 and subsequently transmitted to remote system 120 or transmitted to the remote system and processed by the remote system. A plurality of cameras (not shown) of the visual image system 55 may be positioned to capture different views that an operator would have from within the operator cab 15 of machine 10. In an alternate embodiment, a plurality of cameras (not shown) may be positioned to provide a point of view including the machine 10 and a portion of the work site at which the machine is operating.
An example of a remote system may be a remote control console 130 schematically depicted adjacent instrument array 30 in
Each input device may function as a console input device for providing commands or data input such as inputting information, changing operations, and issuing commands to the machine 10 through the remote control console 130. As with the machine instrument array described above, console instrument array 131 may include a plurality of fixed or stationary input devices such as buttons, knobs, dials, levers, joysticks, and other controls that are permanently mounted the remote control console 130. Fixed function input devices 136 of the remote control console 130 may have a fixed functionality such that they are always used to control the same function. The modifiable function input devices 137 may have a modifiable or changeable functionality such that they may be modified to control the input of different functions. The fixed function input devices 136 and the modifiable function input devices 137 may be similar to the respective fixed function input device 33 and the modifiable function input device 34 as described above.
The point of view display 132 may include one or more displays that are positioned adjacent the console instrument array 131 in a manner as desired such as for efficient operation of the remote control console 130 by an operator. In doing so, the point of view display 132 may include a plurality of displays or areas on a single display upon which visual images are displayed to depict a point of view relative to machine 10. For example, a first point of view display 140 may be centered above the console instrument array 131, and a pair of second point of view displays 141 positioned on opposite sides of the first point of view display. The first point of view display 140 may be used to display images in front of machine 10 while the second point of view displays 141 may be used to display images to the right and left of the machine. If desired, additional images of other points of view may be displayed such as behind the machine 10 or views of other aspects of the machine such as the blade.
In addition to configuring the point of view display 132 in a manner similar to that which an operator would see when operating machine 10, the console instrument array 131 of remote control console 130 may also be configured generally identically to the instrument array 30 within the operator cab 15. By positioning the various display devices and the input devices in generally the same location on the console instrument array 131, the transition of an operator moving from operating machine 10 from within the operator cab 15 to operating machine 10 from remote control console 130 may be simplified. In one example, the console display devices and console input devices of the console instrument array 131 may be generally identical in configuration and position to each of the machine display devices and the machine input devices of instrument array 30. More specifically, the console instrument array 131 may be configured so that it includes gauges 134 and displays 135 that generally match the gauges 31 and displays 32 of the machine instrument array. Similarly, fixed function input devices 136 and modifiable function input devices 137 may be provided on console instrument array 130 so that they generally match the fixed function input devices 33 and the modifiable function input devices 34 of the instrument array 30.
In an alternate embodiment, rather than build console instrument arrays 131 that match the instrument array 30 of each type of machine 10, some or all of the fixed function components of the instrument array 30 may be replaced on console instrument array 131 by images on displays 135 of the console instrument array. For example, controller 51 may generate images on the displays 135 of console instrument array 131 that are generally identical to some or all of the gauges 31 of the instrument array 30. In instances in which the images are not generally identically configured, it may still be desirable to position the images of the gauges 31 on the displays 135 in generally identical positions as compared to those on the instrument array 30.
Similarly, some or all of the fixed function input devices 33 may be replaced by modifiable function input devices 137 on console instrument array 131 in the form of images on a touch screen display or a display associated with another input device. In instances in which the input devices or images thereof are not generally identically configured, it may still be desirable to position the input devices on the console instrument array 131 in generally identical positions to those on the instrument array 30.
The console controller 133 may form a portion of controller 51 to assist in the operation of machine 10. In doing so, the console controller 133 may interact with the machine controller 52 as well as any of the other controllers associated with the machine 10 to control various functions and/or features of the machine 10.
Still further, the remote control console 130 may include a console mounting location 138 for removably mounting the portable computing device 100 thereat. As with the instrument array 30, components of the portable computing device 100 may supplement or replace some of the components of remote control console 130. For example, the display 103 of portable computing device 100 may supplement or replace aspects of the console instrument array 131. In addition, the functionality of controller 51 may be distributed so that certain functions are performed by the machine controller 52 (as well as other controllers on the machine 10), and other functions may be performed by the portable device controller 110 of portable computing device 100, and still other functions may be performed by the console controller 133. As with mounting the portable computing device 100 on the machine 10, sensors, displays and input devices of the portable computing device 100 may supplement or replace some of the sensors, displays, gauges, and input devices of remote control console 130 upon mounting the portable computing device on the remote control console. As a result, upon operatively mounting the portable computing device 100 at the console mounting location 138, the display devices of the console instrument array 131 may be configured to display information related to operation of the machine and the console input devices may be configured to receive input commands from an operator.
Through such a configuration, upon operatively mounting the portable computing device 100 at machine mounting location 109 of machine 10, the controller 51 may operate to include the machine controller 52, any other controllers on machine 10, and the portable device controller 110. The controller 51 may be configured to receive signals from the plurality of sensors of the machine 10 and determine the operating conditions of the machine based upon the signals.
For each operating condition of the machine 10, a predetermined configuration of the instrument array 30 of the machine 10 may be established or determined. If desired, a plurality of possible predetermined configurations may be stored in data maps of the controller 51. Once the operating conditions of the machine are determined, the controller 51 may transmit a plurality of signals (i.e., machine signals) to the machine instrument array indicative of the operating conditions of the machine 10. In instances in which the instrument array 30 includes displays 32, the signals may reflect the image and location of each display as well as the value of the operating condition displayed.
Upon operatively mounting the portable computing device 100 at the console mounting location 138 of remote control console 130, the controller 51 may operate to include the machine controller 52, any other controllers on machine 10, the portable device controller 110, and the console controller 133. As in the situation in which the portable computing device 100 is operatively mounted at machine, the controller 51 may receive signals from the plurality of sensors of the machine 10 and determine the operating conditions of the machine based upon the signals. The controller 51 may also transmit visual image signals from one or more visual image sensors associated with the machine 10, process the signals and display point of view images on the point of view display 132 at the remote control console 130 based upon the visual image signals.
Based upon the operating conditions of the machine 10 and the predetermined configuration of the machine instrument array for those operating conditions when the portable computing device 100 is mounted at machine 10, controller 51 may generate and transmit a plurality of signals (i.e., console signals) to the console instrument array 131. The signals may be indicative of the operating conditions of the machine and define a configuration of the console instrument array so that, for each operating condition, the configuration of the console instrument array is generally identically to the predetermined configuration of the machine instrument array. The controller 51 may further be configured to transmit signals from the remote control console 130 to the machine 10 to direct the machine to perform desired operations.
By configuring the machine 10 and the remote control console 130 to receive a portable computing device 100 having a display 103, the display forms a portion of the machine display of the machine instrument array upon mounting the portable computing device at the machine 10 and forms a portion of the console display of the console instrument array upon mounting the portable computing device at the remote control console 130. The controller 51 is configured to determine a plurality of images to be displayed based at least in part upon a set of operating conditions of the machine 10, and is further configured to display the plurality of images on the machine display upon mounting the portable device display 100 at the machine, and to display the plurality of images on the console display upon mounting the portable device display at the remote control console 130.
In instances in which the instrument array 30 further include a gauge 31 at a predetermined location on the instrument array, the controller 51 may be configured to display an image of a gauge 31 on the console display at a location on the console instrument array 131 generally corresponding to the predetermined location. In instances in which both the instrument array 30 and the console instrument array 131 include a gauge 31 and the console instrument array 131 further includes a console gauge, the machine gauge and the console gauge may be generally identically configured. Still further, in instances in which the instrument array 30 further include an input device, the console instrument array 131 may include a console input device that is generally identical to the machine input device. For example, if each of the instrument array 30 and the console instrument array 131 includes a variable or modifiable function input device, the controller 51 may be configured to determine a plurality of images to be displayed generally adjacent each modifiable function input device based at least in part upon a set of operating conditions of the machine. The controller 51 may be further configured to display the plurality of images generally adjacent the modifiable function input device 34 of the instrument array 30 upon mounting the portable computing device 100 at the machine 10, and to display the plurality of images generally adjacent the modifiable function input device 137 of the console instrument array 131 upon mounting the portable computing device at the remote control console 130.
Additional data codes may include authorization keys so that an operator may only use those machines for which proper authorization has been received. Still further, the data codes may include limited access keys so that only certain types of operations may be performed by an operator or other personnel (e.g., a mechanic) based upon the stored limited access keys.
At node 61, the controller 51 may receive signals from the various sensors associated with various aspects of machine 10 as well as any sensors associated with portable computing device 100. At node 62, the controller 51 may receive signals from various input devices associated with machine 10 and portable computing device 100.
The controller 51 may generate various output signals based upon the data codes received at node 60, the sensor signals received at node 61, and the signals from the input devices received at node 62. At node 65, the controller 51 may generate output signals that are transmitted to the gauges 31. In other words, the controller 51 may generate signals that are transmitted to the display devices in which the function is fixed. At node 66, the controller 51 may generate signals or images that are transmitted to the displays 32. More specifically, based upon the user preferences, the operating conditions of the machine 10, and any input commands from an operator, the controller 51 may determine the information to be displayed on the displays 32. In doing so, the controller 51 may also determine which images should be displayed on the display 103 of the portable computing device 100. At node 67, the controller 51 may generate signals or images that are transmitted to displays associated with the modifiable function input devices 34 including any associated with portable computing device 100. At node 68, the controller 51 may generate operating commands to control the operation of machine 10.
At node 61, the controller 51 may receive signals from the various sensors associated with various aspects of machine 10 as well as any sensors associated with portable computing device 100. At node 62, the controller 51 may receive signals from various input devices associated with remote control console 130 and portable computing device 100. At node 63, the controller may receive signals from visual image system 55 indicative of a point of view relative to the machine 10.
The controller 51 may generate various output signals based upon the data codes received at node 60, the sensor signals received at node 61, the signals from the input devices received at node 62, and the visual image signals received at node 63. At node 65, the controller 51 may generate output signals that are transmitted to the gauges 134 of the remote control console 130. In other words, the controller 51 may generate signals that are transmitted to the display devices in which the function is fixed. At node 66, the controller 51 may generate signals or images that are transmitted to the displays 135. More specifically, based upon the user preferences, the operating conditions of the machine 10, and any input commands from an operator, the controller 51 may determine the information to be displayed on the displays 135 of remote control console 130. In addition, the controller 51 may determine which images should be displayed on the display 103 of the portable computing device 100. At node 67, the controller 51 may generate signals or images that are transmitted to displays associated with the modifiable function input devices 137 including any associated with portable computing device 100. At node 68, the controller 51 may generate operating commands to control the operation of machine 10. At node 69, the controller 51 may generate images to be displayed on the point of view display 132.
At stage 163, the controller 51 may receive data from the various sensors associated with the machine 10 and portable computing device 100. At stage 164, the controller 51 may determine the operating conditions of the machine 10 based upon the data received from the sensors at stage 163. At stage 165, the controller 51 may determine based at least in part on the operating conditions of the machine which images should be displayed on the displays 32 of the machine and which images should be displayed on the display 103 of the portable computing device 100. As such, the instrument array 30 may have a predetermined configuration for each operating condition.
More specifically, for some operating conditions, the predetermined configurations may be identical and for other operating conditions, the predetermined conditions may be different. As an example, when machine 10 is performing certain tasks or operations, it may be desirable to show certain operating conditions of the machine 10 such as the pressure associated with certain of the hydraulic cylinders of the machine. In other situations, it may be desirable to depict the position of blade relative to a ground reference. At stage 166, the controller 51 may transmit a plurality of machine signals to the instrument array 30 to configure the instrument array so that it will reflect the operating conditions of the machine 10.
If any of the input devices are configured as modifiable function input devices 34, the controller 51 may determine at stage 167 the function of each modifiable function input device. In addition, the controller 51 may generate at stage 168 a signal to display an image generally adjacent to such modifiable function input devices 34 so as to communicate the functionality to an operator within operator cab 15.
At stage 169, the controller 51 may receive various input signals from an operator. At stage 170, the controller may generate operating commands to operate the machine 10 as desired based upon the data codes, signals received from the sensors, and the input from the operator. At stage 171, the controller 51 may transmit the operating commands to the various machine components to operate machine 10 as desired.
If the portable computing device 100 was not removably mounted or docked at machine 10 at decision stage 161, the controller 51 may determine at decision stage 172 whether the portable computing device 100 is docked at the remote control console 130. If the portable computing device 100 is not docked at either the machine 10 or the remote control console 130, the controller 51 may enter a standby mode in which it waits for the portable computing device to be docked in either the machine 10 or the remote control console 130.
If the portable computing device 100 is removably mounted or docked at the remote control console 130, the machine 10 may be operated from the remote control console in generally the same manner as if the operator were located in the operator cab 15. In doing so, it may be desirable for the console instrument array 131 to generally be configured in the same manner as the instrument array 30 of machine 10 and for the console instrument array 131 to operate generally in the same manner as instrument array 30. Upon mounting the portable computing device 100 at remote control console 130, data codes may be transferred at stage 173 from the portable computing device to the console controller 133. In an alternate embodiment, the data codes may be transferred to other portions of controller 51.
At stage 174, the controller 51 may receive data from the various sensors associated with the machine 10 and portable computing device 100. At stage 175, the controller 51 may determine the operating conditions of the machine 10 based upon the data received from the sensors at stage 163. At stage 176, visual image signals indicative of the point of view relative to the machine 10 may be transmitted from machine 10 to the remote control console 130. At stage 177, point of view images may be generated and displayed on the point of view display 132 based upon the visual image signals received from the visual image system 55 on machine 10.
At stage 178, the controller 51 may determine based at least in part on the operating conditions of the machine 10 which images should be displayed on the displays 135 of the remote control console 130 and which images should be displayed on the display 103 of the portable computing device 100. As such, the instrument array 30 may have a predetermined configuration for each operating condition. In one embodiment, it may be desirable for the images to be displayed on the displays 135 of the remote control console 130 to be generally identical to those displayed on the displays 32 of the machine 10 for the same set of operating conditions. In other embodiments, the images displayed on the displays 135 may not be generally identical to those displayed on displays 32 of machine 10 but it may still be desirable for the images to be in generally identical positions. In that manner, it may be easier for an operator familiar with operating machine 10 to operate the machine through remote control console 130. In other words, if an operator is familiar with the configuration and location of the various displays on the machine 10, remote operation of the machine will likely be simplified if the various displays are similarly configured and located on the remote control console 130. In the absence of similarly configured images, remote operation may still be simplified if the location of the images remains generally consistent.
At stage 179, the controller 51 may transmit a plurality of console signals to the console instrument array 131 to configure the console instrument array so that it will reflect the operating conditions of the machine 10.
If any of the input devices are configured as modifiable function input devices 137, the controller 51 may determine at stage 180 the function of each modifiable function input device. In addition, the controller 51 may generate at stage 181 a signal to display an image generally adjacent to such modifiable function input devices 137 so as to communicate the functionality to an operator at the remote control console 130.
At stage 182, the controller 51 may receive various input signals from an operator. At stage 183, the controller may generate operating commands to operate the machine 10 as desired based upon the data codes, signals received from the sensors, and the input from the operator. At stage 184, the controller 51 may transmit the operating commands from the remote control console 130 to the machine controller 51 to operate the various machine components of machine 10 as desired.
Industrial Applicability
The industrial applicability of the system described herein will be readily appreciated from the foregoing discussion. The foregoing discussion is applicable to machines that may be operated by remote control. Machine 10 and remote control console 130 may each be configured to receive a portable computing device 100 thereat. In one embodiment, each of the machine 10 and the remote control console 130 may be inoperable without the portable computing device 100.
The instrument array 30 of the machine 10 and the console instrument array of the remote control console 130 may be generally identically configured. In one embodiment, each of the display devices and input devices may be identically configured and generally identical images displayed for each operating condition of machine 10. In another embodiment, displays may be used to display images that are generally identical to gauges and input devices of the other the instrument array. In still other embodiments, the gauges, input devices and images display may not be generally identical from one instrument array to the other but it may still be desirable for the images to be in generally identical positions. Such a configuration may simplify the transition of a machine operator from operating a machine 10 from within the operator cab 15 to operating the machine via remote control.
It will be appreciated that the foregoing description provides examples of the disclosed system and technique. However, it is contemplated that other implementations of the disclosure may differ in detail from the foregoing examples. All references to the disclosure or examples thereof are intended to reference the particular example being discussed at that point and are not intended to imply any limitation as to the scope of the disclosure more generally. All language of distinction and disparagement with respect to certain features is intended to indicate a lack of preference for those features, but not to exclude such from the scope of the disclosure entirely unless otherwise indicated.
Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.
Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.
Number | Name | Date | Kind |
---|---|---|---|
4104592 | Ratz et al. | Aug 1978 | A |
4855822 | Narendra et al. | Aug 1989 | A |
5054569 | Scott et al. | Oct 1991 | A |
5399844 | Holland | Mar 1995 | A |
5448479 | Kemner et al. | Sep 1995 | A |
5459660 | Berra | Oct 1995 | A |
5469356 | Hawkins et al. | Nov 1995 | A |
5551524 | Yamamoto et al. | Sep 1996 | A |
5646843 | Gudat et al. | Jul 1997 | A |
5680328 | Skorupski et al. | Oct 1997 | A |
6112139 | Schubert et al. | Aug 2000 | A |
6182006 | Meek | Jan 2001 | B1 |
6202008 | Beckert et al. | Mar 2001 | B1 |
6246935 | Buckley | Jun 2001 | B1 |
6429773 | Schuyler | Aug 2002 | B1 |
6430485 | Hullinger | Aug 2002 | B1 |
6633800 | Ward et al. | Oct 2003 | B1 |
6663010 | Chene et al. | Dec 2003 | B2 |
6778097 | Kajita et al. | Aug 2004 | B1 |
6782644 | Fujishima et al. | Aug 2004 | B2 |
7032703 | Wulfert et al. | Apr 2006 | B2 |
7050795 | Wiegand et al. | May 2006 | B2 |
7565941 | Cunningham | Jul 2009 | B2 |
7578079 | Furem | Aug 2009 | B2 |
7742861 | Lee | Jun 2010 | B2 |
7894951 | Norris et al. | Feb 2011 | B2 |
8051936 | Hebert et al. | Nov 2011 | B1 |
8106757 | Brinton et al. | Jan 2012 | B2 |
8139108 | Stratton et al. | Mar 2012 | B2 |
8237389 | Fitch et al. | Aug 2012 | B2 |
8272467 | Staab | Sep 2012 | B1 |
8918230 | Chen et al. | Dec 2014 | B2 |
8977407 | Dorneich et al. | Mar 2015 | B2 |
20010037163 | Allard | Nov 2001 | A1 |
20030036817 | Morley et al. | Feb 2003 | A1 |
20030147727 | Fujishima et al. | Aug 2003 | A1 |
20030171863 | Plumeier et al. | Sep 2003 | A1 |
20060106510 | Heffington | May 2006 | A1 |
20060195483 | Heider et al. | Aug 2006 | A1 |
20060224280 | Flanigan et al. | Oct 2006 | A1 |
20060271251 | Hopkins | Nov 2006 | A1 |
20060271263 | Self et al. | Nov 2006 | A1 |
20070143482 | Zancho | Jun 2007 | A1 |
20080103640 | Watanabe et al. | May 2008 | A1 |
20080172147 | Taki et al. | Jul 2008 | A1 |
20090102626 | Lesesky | Apr 2009 | A1 |
20100036560 | Wright et al. | Feb 2010 | A1 |
20100179844 | LaFergola et al. | Jul 2010 | A1 |
20100185638 | Wright et al. | Jul 2010 | A1 |
20100249957 | Price | Sep 2010 | A1 |
20110257816 | Song et al. | Oct 2011 | A1 |
20110282519 | Carlsson | Nov 2011 | A1 |
20120136525 | Everett et al. | May 2012 | A1 |
20120209634 | Ling et al. | Aug 2012 | A1 |
20120229662 | Lankalapalli et al. | Sep 2012 | A1 |
Entry |
---|
Teleoperated Bulldozer, ASI Robots, Uploaded Mar. 3, 2010, https://www.youtube.com/watch?v=S-mgj8cZTNk. |
Meehan, Christopher J., A remote control system for teleoperation of a skid-steer loader over a mobile Wi-Fi mesh network, Colorado School of Mines, M.S. Thesis, Spring 2011. |
Kwitowski, August J., etc., Teleoperation for Continuous Miners and Haulage Equipment, IEEE Transactions on Industry Applications, vol. 28, No. 5, Sep./Oct. 1992, pp. 1118-1125. |
Hainsworth, D.W., Teleoperation User Interfaces for Mining Robotics, Autonomous Robots 11, pp. 19-28, 2001. |
Coal News, vol. 8, No. 10, Oct. 2011. |
Ilstam, Anders, Sandvik Mining and Construction, Mar. 11, 2002. |
Wohlford, William P., etc., New capability for remote controlled excavation, 1989 Proceedings of the 6th ISARC, pp. 33-40, 1989. |
Henderson, Steven J. et al., “Augmented Reality for Maintenance and Repair (ARMAR),” Air Force Research Laboratory, Wright-Patterson AFB, Aug. 2007, 70 pp. |
Number | Date | Country | |
---|---|---|---|
20140172128 A1 | Jun 2014 | US |